My save files are getting bigger and bigger

[olight]

Currently my game is on 1000+ cycles, it's so laggy and barely playable. I'd checked my save files and it seems as cycles passed, the save files are getting bigger. But i have no idea why my save files are getting bigger and bigger.

Here is the size of my several save files, each of them of different worlds:

Cycles	Dupes	Size(Bytes)
1041	15	31,959,235
375	14	14,969,805
112	8	7,752,485
64	4	4,928,762
21	4	3,643,498
16	3	4,580,577
14	4	3,170,540
8	4	3,174,607
1	3	2,425,710
1	3	2,505,445
1	3	2,484,618

 

[AlexRou]

The more changes you make to the world the bigger it gets cuz the more changes and states of things it needs to store. But the size of the save file is not related to the lag, the lag is cuz of how many dupes and buildings you have.

[dwarftopia]

11 hours ago, olight said:

Currently my game is on 1000+ cycles, it's so laggy and barely playable. I'd checked my save files and it seems as cycles passed, the save files are getting bigger. But i have no idea why my save files are getting bigger and bigger.

Here is the size of my several save files, each of them of different worlds:

Cycles	Dupes	Size
1041	15	31,959,235
375	14	14,969,805
112	8	7,752,485
64	4	4,928,762
21	4	3,643,498
16	3	4,580,577
14	4	3,170,540
8	4	3,174,607
1	3	2,425,710
1	3	2,505,445
1	3	2,484,618

 

Are those values in bytes or kilobytes?

[Lifegrow]

bytes,,,

I've lots of 350-400ish saves (normally the point i restart) that are 15-30meg

[olight]

2 hours ago, dwarftopia said:

Are those values in bytes or kilobytes?

Bytes. 

[AlexRou]

The only time save files have anything to do with performance is that a lazily implemented save system will freeze the game while saving if it takes too long to generate the save file or the save file is too big. The game only autosaves once per cycle so the size of which has nothing to do with your lags.

Go open the task manager and look at CPU usage making sure to enable one graph per core, if even once CPU core is anywhere near 100% that will explain your lags. In that case, the lag is being caused by having too many dupes/buildings/piping and maybe even too many materials on the ground.

[Beowulfe]

36 minutes ago, AlexRou said:

The only time save files have anything to do with performance is that a lazily implemented save system will freeze the game while saving if it takes too long to generate the save file or the save file is too big. The game only autosaves once per cycle so the size of which has nothing to do with your lags.

Go open the task manager and look at CPU usage making sure to enable one graph per core, if even once CPU core is anywhere near 100% that will explain your lags. In that case, the lag is being caused by having too many dupes/buildings/piping and maybe even too many materials on the ground.

It may also indicate that world objects are building up over time, causing eventual slowdown - the objects themselves may not be a cause of concern, but could definitely be a good first point to look for optimization on games that have been running for a large number of cycles.

[Risu]

Got an example of one of these bloated saves?
 

[Zervo]

 

19 hours ago, AlexRou said:

The more changes you make to the world the bigger it gets cuz the more changes and states of things it needs to store. But the size of the save file is not related to the lag, the lag is cuz of how many dupes and buildings you have.

but still, IMO 30mega is a bit too much though.

surely they should try to compress it a bit more.

perhaps there are data/records that aren't needed anymore and still being store, or just make an overhaul change about how they store the save file

 

large save file will make the save/load time longer.

[olight]

22 hours ago, olight said:

 

Cycles	Dupes	Size(Bytes)
1041	15	31,959,235

 

1101 Cycles, 15 dupes, 32,911,038 Bytes now.

I'd tried to reduce lags of game and size of save file. Minimized digging, swept almost materials, kept the numbers of dupes, even not explored other area for a several hundred cycles. Recently i'd been trying to change all polluted oxygen into pure oxygen. Polluted oxygen was spreaded largely through entire world and there are some high densed polluted area yet.

Anyway regardless all my efforts, the size of save file is getting bigger.

For anyone who interested, here is my "Big" save file of "Small" colony. 

The Outrageous Pigpen Cycle 1101.zip

[Risu]

17 minutes ago, olight said:

1101 Cycles, 15 dupes, 32,911,038 Bytes now.

I'd tried to reduce lags of game and size of save file. Minimized digging, swept almost materials, kept the numbers of dupes, even not explored other area for a several hundred cycles. Recently i'd been trying to change all polluted oxygen into pure oxygen. Polluted oxygen was spreaded largely through entire world and there are some high densed polluted area yet.

Anyway regardless all my efforts, the size of save file is getting bigger.

For anyone who interested, here is my "Big" save file of "Small" colony. 

The Outrageous Pigpen Cycle 1101.zip

1045 Tiles: 2,154,069 bytes
1835 Insulated Tiles: 3,881,457 bytes
2939 Ladders: 6,042,584 bytes
1429 Heavy Wires: 2,892,934 bytes
1188 Liquid Pipes: 2,408,229 bytes
2235 Gas Pipes: 4,530,498 bytes
844 Insulated Liquid Pipes: 1,732,885 bytes
 

[olight]

3 minutes ago, Risu said:

1045 Tiles: 2,154,069 bytes
1835 Insulated Tiles: 3,881,457 bytes
2939 Ladders: 6,042,584 bytes
1429 Heavy Wires: 2,892,934 bytes
1188 Liquid Pipes: 2,408,229 bytes
2235 Gas Pipes: 4,530,498 bytes
844 Insulated Liquid Pipes: 1,732,885 bytes
 

How did you get these numbers? Sum of that is 23,642,656 Bytes. At starting a world, the save file size is 2-3meg with small explored area. So if added explored area, It seems right that buildings are related to the save file size.

[Risu]

Just now, olight said:

How did you get these numbers? Sum of that is 23,642,656 Bytes. At starting a world, the save file size is 2-3meg with small explored area. So if added explored area, It seems right that buildings are related to the save file size.

I've got the whole save mapped out. Just have to read it. The simulator side amounts to 2,085,028 bytes by the way.
 

[olight]

7 minutes ago, Risu said:

The simulator side amounts to 2,085,028 bytes by the way.
 

 It's not as influential as i thought.

[AlexRou]

4 hours ago, Zervo said:

 

but still, IMO 30mega is a bit too much though.

surely they should try to compress it a bit more.

perhaps there are data/records that aren't needed anymore and still being store, or just make an overhaul change about how they store the save file

 

large save file will make the save/load time longer.

A normal HDD will have reads of at least 80 MB/s, if you want to complain about taking half a second (or at most a second if other programs are also doing some minor reading/writing) to read/write the save file then ...

Reducing the size of the save file by any means would result in taking a longer time to process the file. Adding something like compression may make the file smaller but result in using time to compress/decompress and thus net change in save/load time may actually be longer. This also applies to storing just enough data to recreate everything, the more the game needs to process the save file the longer loading, and possibly saving, will be.

It is a fine tuned balancing act, the more data you store the faster it is to process, but the longer it takes to read. Also they would have to take into consideration possible gains vs amount of time spent developing it. At this point 30 MB is still pretty good and doesn't cause major problems.

But still, the size of the save file has nothing to do with the OP's lag.

[Zervo]

1 hour ago, AlexRou said:

A normal HDD will have reads of at least 80 MB/s, if you want to complain about taking half a second (or at most a second if other programs are also doing some minor reading/writing) to read/write the save file then ...

 

Spoiler

 

The thing is, saving/loading is different from copy-paste 1 file in a drive.

You are not just copying 1 whole file from 1 location to other location.
Depends how the program works, i'm not sure how much relevant is that 80MB/s.

 

I figure since you know this much, you surely know that when you copy 1 file with 30MB and 100 files with total 30MB, it took longer to complete the 100 files, highly possible it will pass the 1 second duration.

I don't know and never check how they store their game save file, but probably they store tile by tile similar like block by block (in files/harddrive).

 

And about the compression, I think you misunderstood my post, perhaps because my poor word selection and phrasing too.

I don't mean compress like zip file, but more like how they store the save file/code.

For example (disclaimer: Again I don't really know how it works here, this is just pure example for what i mean by compression):

Like if they store the untouched area (still in a fog) by writing/putting a certain code. for example "00" code in each tile block.

They could change it with "{0" to start the fog area and close it with "0}", and put a number in between to store how many tiles it meant to be, and if it's below 3, they will use regular mechanic (again, this is just an example).

The impact for the save file will be something like this:

Before:

00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00 00
00 00 00 X0 10 0X 01 00 00

00 XX 10 X0 0X 1X 11 X0 0X

After:

{0 07 0}
{0 07 0}

{0 07 0}
{0 03 0} X0 10 0X 01 00 00

00 XX 10 X0 0X 1X 11 X0 0X

Once again, this is just for example purposes, the actual won't be as simple as this.

Please don't take it without any grain. you can put the salt aside tho.

 

1 hour ago, AlexRou said:

But still, the size of the save file has nothing to do with the OP's lag.

 

Spoiler

 

and yes, I don't mean that OP's lag is cause by the file size.

well, not directly at least. I don't think I said it does in my prev post :/

 

Indirectly it might be related, because larger file size meant there are more things to calculate by the computer.

this is why bigger save files indirectly means the game could be more laggy.

 

Regarding performance issue, this is something the dev will have to look into in the future.

Sooner or later, improvement will be needed.

 

I won't be surprise if they will put invisible fog to tag certain tile/area as idle state and by doing this, this will lower the background compute.

Something similar to rendering map/area in open world games. Though I know it's different but the same approach could still be use

 

 

The final post is longer than I ever intended it to be.

Just to make it clear, I'm not replying for the sake of arguing who is right/wrong.

I totally agree with you that OP's lag is not directly related to how big the save file are

[AlexRou]

1 hour ago, Zervo said:

The thing is, saving/loading is different from copy-paste 1 file in a drive.

You are not just copying 1 whole file from 1 location to other location.
Depends how the program works, i'm not sure how much relevant is that 80MB/s.

 

I figure since you know this much, you surely know that when you copy 1 file with 30MB and 100 files with total 30MB, it took longer to complete the 100 files, highly possible it will pass the 1 second duration.

I don't know and never check how they store their game save file, but probably they store tile by tile similar like block by block (in files/harddrive).

The extremely lazy way would yes be writing one chunk of information a time. But anyone at the professional level will make it so that the entire save file is generated in RAM and then written out to the HDD in one go, which is about the same as copying one large file. Also writing 100000 times to a single file is not the same as copying 100000 files. So your point about it being much slower is irreverent. (The game puts all the information it needs into a single file)

Go learn a programming language (a compiled language like c/c++) and write a benchmark test if you want to prove that there is a situation where writing to a single file would take significantly more than a second for a file size of 30 MB where there is low disk usage by other programs. Also there should be nothing other than writing done since we only care about the time it takes to write to disk and not the processing time to generate the data.

Even if you are able to get it to take much longer than a second to write the file, I doubt anything you are doing is something any sane programmer would do in a production environment.

1 hour ago, Zervo said:

Like if they store the untouched area (still in a fog) by writing/putting a certain code. for example "00" code in each tile block.

The way you described it, would mean every time you load, the game has to re generate the entire unexplored portion of the map. So yes you store less but you lose time generating the map again. The game could dynamically generate the map as you explore but why waste the effort implementing it, also this will cause freezing when exploring unless they have a very simple map generation system.

 

Sorry that last line wasn't directed specifically at you. Anyways ...

1 hour ago, Zervo said:

Indirectly it might be related, because larger file size meant there are more things to calculate by the computer.

this is why bigger save files indirectly means the game could be more laggy.

Larger file size means there is less to calculate because there is more pre-calculated data so less processing. If you meant there is more things to calculate as the game is running, nope, it doesn't make sense to do more calculations than you need to run the game. If you already have all the information you need to run the game, then there isn't any more information you need to save the game.

 

 

[olight]

3 hours ago, AlexRou said:

Larger file size means there is less to calculate because there is more pre-calculated data so less processing. If you meant there is more things to calculate as the game is running, nope, it doesn't make sense to do more calculations than you need to run the game. If you already have all the information you need to run the game, then there isn't any more information you need to save the game.

I think most of the save files is data of materials and energy, which is needed for calculation. Mass, heat, electro energy, moving contents in pipes and so on, all of these are the input data and the game calculates mutual behavior of all these things In Real Time. If there is much things to calculate, the game would eat much of the system resources.

Save file is a snapshot of current state of the game. It doesn't mean the calculation is completed but means just paused, so all data is required continuously and remained in the save file.

That's why more builds increase the save file size and big save files cause game lag.

Just my thought.

[AlexRou]

1 hour ago, olight said:

I think most of the save files is data of materials and energy, which is needed for calculation. Mass, heat, electro energy, moving contents in pipes and so on, all of these are the input data and the game calculates mutual behavior of all these things In Real Time. If there is much things to calculate, the game would eat much of the system resources.

Save file is a snapshot of current state of the game. It doesn't mean the calculation is completed but means just paused, so all data is required continuously and remained in the save file.

That's why more builds increase the save file size and big save files cause game lag.

Just my thought.

I think you misunderstood when I said there is less to calculate, I meant there is less to calculate to restore the game state.

You are lagging because there is a lot to calculate in real time not because the save file is big. A big save file does nothing to real time performance. The only time it affects you at all is when saving/loading and that is not being done every single frame.

 

The only reason I think you would think that the size have anything to do with your lag is if the game reads and writes to the save file in real time. It does not do that, no game does that, it reads the file once and never touch it again.