Archived

This topic is now archived and is closed to further replies.

Please be aware that the content of this thread may be outdated and no longer applicable.

JRup

Savefile expansion knowledge needed

Recommended Posts

JRup    164

Hi!
 

I'm hoping some help or insight can be gained from this thread, so here goes:

We already know that save times tend to grow as the map is uncovered, contraptions are built and so on... This means that as data accumulates it shall be sent to the save file.... Obvious right? I'm more interested in something that happens in between and this is where I'd like to know if it's worth chasing...

My current save file is 28.5MB in size, rather small when we look at current drive sizes.

As any curious keyboard warrior would, I'd been suspecting save files can't be so small without some secret sauce...

Turns out I'd looked into this subject and at most there's one post in this forum that wanted more info about the data format inside the saves... Go further down the rabbit hole and you bump into duplicity... This means cat is out of the hat and we can now edit some save file information with the risks that entails. I'm not interested in playing with edited savefiles, however.  The knowledge that the save file is compressed is the apple brought in by the snake though.

So, after further testing I see this there:

 

dup.png.6d933d54b8e07c3367bcc733f909de59.png

 

May I direct your attention to the "isCompressed" checkbox? This is the whole point of this post.

After unmarking the checkbox, 28.5 MB will balloon into about 145 MB, a factor of 5 if we round up.

Disregarding any other possible changes one could do to the file, ONI will merrily load save and it will chug along just like any other day, but by the end of a cycle or a normal save we're back at the 28.5 MB... So compression kicks in automagically.

Debug mode is easily enabled but I see it as just a beefed up sandbox mode. I haven't found any options regarding savefile compression. I also haven't found any plain text file that addresses this option.

A consequence of being able to "fly" uncompressed is that all save files would gobble up disk space, but this is not the focus of the post. I'm managing my computer just fine and if it "bursts into flames" I'm the one that has to run (a little exaggerated - issue would be just disk space but you understand - no brooms and mops will go out of control with this magic; the buckets were innocent).

So here are some questions to our dear forum dwellers:

Does anyone have a way to just disable automagic compression? I'd like to know if this could enhance save times... Of course, my pc is not in the ultra e-p**n brag category, I would like to state that it's a regular box that handles saving just ok... But the grass is greener on the other side.

Has anyone had prior experience with this and be able to say performance saving through this is just meh?

I beseech your arcane knowledge.

 

 

 

 

Share this post


Link to post
Share on other sites
SharraShimada    914

The simulation pauses when the game makes a savefile. So there is no benefit of saving umcompressed i think, because the CPU is already free to do the work of compressing.  

Today the system can compress from RAM directly to a file on the disk. So you have to choose: Save hundreds of MB directly to the disk, or compress it on the fly, and store way less data. Compression takes CPU-time, data-storage consumes time for writing to the disk. 

In theory, a raw storage without compression would get you ca small performance gain, with a fast SSD, but not that much as you would think. We are talking about merely 1 or 2 seconds... The downside is a vast usage of disk space. 

Share this post


Link to post
Share on other sites
JRup    164

I absolutely hate load and save times in any game. I was spoiled by the NES, hated the PS1 for that reason and still bemoan any company that includes shoddy optical hardware in their consoles. But I digress.

51 minutes ago, SharraShimada said:

the CPU is already free to do the work of compressing.

This is nevertheless an overhead.  And IMO the game then turns into a compression utility, and it pauses to actually do so.

I currently can't say that if the save files were not compressed by the game then they'd be 5 times as big. I can only speculate on the actual size given what duplicity can provide: 30MB turn into 150MB by my grossly exaggerated estimate.

ONI only preserves 5 autosaves so then we're rounding up to at least 1GB disk space for save files (if uncompressed).... Not to mention manual saves... My save folder currently rounds up to 2GB because I'm a digital hoarder. We can track the base I'm building from start to where it is today. It would make 10GB "uncompressed".

SSDs are no longer the novelty and are pretty much pushing out HDDs as the system drive. HDDs have never stopped being cost effective storage wise though.

SSDs are actually being replaced by NVME drives much to the point that I believe I'd rather have the pc write 100-200MB than wait for compression, and this shows: threads for ONI lag related subjects are starting to pop up... I just hope that it doesn't do a full chia pet bloom.  Nowadays we have to build against performance/compression- killer aspects of the game. When I take a screenshot I don't mind showing my clutterbug practices :D (But I know I'll have to sweep it all eventually.)

Will this aspect of the game turn it into "can it handle an ONI save file" much like a "can it run cry*is"?

Share this post


Link to post
Share on other sites
ExEvolution    26

@JRup NVME drives are still SSDs, they just run on a newer communication standard, Non-Volatile Memory Express, which uses PCI-E lanes on your CPU to provide more bandwidth for drives, and NVME also greatly increases the disk queue depth from 32 to 65,535 queues, meaning more parallelization is possible, and applications have to wait less time before they can add their request to the disk queue on a busy system. NVME has no noticeable impact on gaming performance or saving/loading, there just isn't enough disk activity for it to matter. SSD vs HDD however makes a huge difference.

Share this post


Link to post
Share on other sites
JRup    164
3 minutes ago, ExEvolution said:

NVME drives are still SSDs

Yes, but actually.. oh well.. Yes it's storage in chips, but now we're looking at SATA limitations vs PCIe limitations... Right now I'm just stating that even SATA could be more than enough to disregard the need to compress the save file.

What would be really great is if the game actually had a hidden option to pass either via steam commands or config file to disable save file compression and see any "performance gain" for ourselves. Let's dream a little further and have someone mod an option into the menus like "Save file compression ON/OFF"

 

Share this post


Link to post
Share on other sites
Gurgel    2165

Well, saving is fast for ONI. And that is where compression is slow, so it hardly matters. What is slow is loading, and there compression is fast, again so it hardly matters. 

Share this post


Link to post
Share on other sites
Cipupec2    47
4 hours ago, SharraShimada said:

The simulation pauses when the game makes a savefile. So there is no benefit of saving umcompressed i think, because the CPU is already free to do the work of compressing.  

Today the system can compress from RAM directly to a file on the disk. So you have to choose: Save hundreds of MB directly to the disk, or compress it on the fly, and store way less data. Compression takes CPU-time, data-storage consumes time for writing to the disk. 

In theory, a raw storage without compression would get you ca small performance gain, with a fast SSD, but not that much as you would think. We are talking about merely 1 or 2 seconds... The downside is a vast usage of disk space. 

I don't know how ONI does things, but IF the simulation is paused until saving is done (i.e. saving is not done in the background) then wouldn't the system be IO bound during that time? Because CPU is MUCH faster, on orders of magnitude, even compared to the most advanced type of storage interface. 

Also to my [limited] knowledge the few realtime strategy simulation games with frequent saving I know do use compression.

Share this post


Link to post
Share on other sites
JRup    164
11 minutes ago, Cipupec2 said:

wouldn't the system be IO bound?

I wouldn't say as much, the pause does indeed help with the unwanted problem that comes when building anything during a save... (dragging wires/pipes, anyone?) But all game progress should be in memory and the compression should also take place in memory; finally, writing 30MB (or 200MB) shouldn't take so long.

Share this post


Link to post
Share on other sites
TheMule    344
17 hours ago, JRup said:

This is nevertheless an overhead.

So is the extra allocation needed for a larger file. Finding a place (blocks, clusters, whatever your filesystem calls them) for 150MB takes longer than for 30MB, not to mention the actual transfer.

In today's world, CPU time is extra cheap, unless you're already at 100% on all of your cores (which is something ONI doesn't do). If even as much as one core is free, that's way enough computing power to compress data and generate a stream of bytes at a rate larger than the bandwidth of the fastest SSD. And it's not going to slow down (significantly) what's happening on the other cores.

 

Share this post


Link to post
Share on other sites
JRup    164
3 hours ago, TheMule said:

Finding a place (blocks, clusters, whatever your filesystem calls them) for 150MB takes longer than for 30MB, not to mention the actual transfer.

Then, what happens when storage speed catches up? I just did some (not so scientific) tests on my computer. Warning, innocent bytes were harmed in the process.

#1 Found a 358MB iso, copied, then pasted. Result: Instant duplicated data generation. Did not even take a second.

Test #2 copied same file to an external usb 3 NVME enclosure, virtually same result.

Test #3 renamed iso inside enclosure and re-copied back to system. Virtually same result.

Results for tests 1-3 no appreciable transfer time needed.

Next, use a bigger iso and re-test. (2.8GB)

Test #4 copy on local machine: 2 seconds tops: no "transfer bar" generated.

Test #5 Copy to external media: several seconds no more than 15s. Estimated speed 219 MB/s (not bits, bytes)

image.png.b9de5acd7da5f07ece029ab89ca95b33.png

Test #6 Rename and copy back to pc: same result as test #4.

So tests 5 and 6 are somewhat interesting and have to do with the fact that even though the external NVME is fast, the storage in this pc is actually faster because it's a RAID array made with the same type of drives and can handle the file better.

What I'm getting at is that when we're able to disregard the speed of storage then we're able to see where the overhead is. Right now it I'm speculating that it is in the compression stage. This is on a subjective observation that my setup can actually handle 30MB like skittles.

3 hours ago, TheMule said:

In today's world, CPU time is extra cheap, unless you're already at 100% on all of your cores (which is something ONI doesn't do). If even as much as one core is free, that's way enough computing power to compress data and generate a stream of bytes at a rate larger than the bandwidth of the fastest SSD. And it's not going to slow down (significantly) what's happening on the other cores.

And the compression of the game information will take the main stage regardless of CPU time availability. The game is paused, the information in memory is read, compressed and finally written to disk. I would dare say this sequence is synchronous with game information hence the pausing of the game and the wait time.

If it could be done asynchronously, then I suspect a little extra memory would be used to leave a snapshot to work on - compress that info on another thread/core then write to disk. All done in the background. No pausing. More dev time needed pushing precious DLC down the line, I surmise.

Let's not discuss power losses or interrupting the save process for some other reason.

I'm just looking for a way to just disable compression in a way that may be there from the beginning. If it has been coded in and if there is a hidden configuration option then that'd be a great find for me. If it's not there, then boohoo but we all had fun with the discussion anyway.
 

Share this post


Link to post
Share on other sites
TheMule    344
13 minutes ago, JRup said:

I just did some (not so scientific) tests on my computer.

You need to either disable OS buffering or to write something that is many times your available RAM, or to ensure "data on platter" via OS calls, otherwise you're getting mixed results with most data being still in RAM. All modern OSes delay writing to disk to some extents, unless told otherwise. 

Share this post


Link to post
Share on other sites
JRup    164
1 hour ago, TheMule said:

All modern OSes delay writing to disk to some extents, unless told otherwise. 

Hence the tests where copy to and from external media were done. Linux and floppy disks was such a joyous combination in its day.

Still, 30MB skittles.

Share this post


Link to post
Share on other sites
KittenIsAGeek    1678

So.....

Which is faster: Compressing data by 5x and writing a file that is 1/5th the size, or writing the original file.

 

Lets say that your CPU runs at 2.5 GHz.  That's 2.5x10^9 "actions" per second that can be taken by the processor.  Lets assume that your CPU is "old" and and all actions follow the "command, then action" method.   Cycle 1: Get next command. Cycle 2: Perform command. Cycle 3: Get next command, etc.  That means your actual throughput is 1.25x10^9 "actions" per second.   I'm going to continue to dramatically simplify things with the next assumption: To compress data, your CPU will take exactly one cycle to compress the data it has grabbed from memory (this is not the actual case, but we'll use it for our hypothetical simulation).  We're left with the following actions:  

  1. Fetch data from memory
  2. Compress data
  3. Write data to disk.

With these assumptions,  we're down to 4.16x10^8 'compressions' per second.  If we eliminate the 'Compress data' piece from our algorithm above, we can write directly to the disk 6.25x10^8 times per second.  So the difference is 2 'actions' vs 3 for compression.  Looks like uncompressed is winning for the moment.  Lets use the numbers for the save file size that were posted earlier: 28.5mb and 145mb.  Lets also assume that our computer has a 64-bit CPU.  28.5 megabytes is 228 megaBITs as 1 byte is 8 bits.  145mb is 1160 megaBITs.  

Before we continue, lets make one more assumption that will be critical.  For the next bit, we're assuming that a write to a disk takes exactly as long as a read from memory.  This is NOT true in almost every possible case, but we're going to pretend it is for a bit.

Lets assume that our memory size is 1160 megaBITs in size and that the uncompressed file is simply a direct copy (probably not the actual case).  Since we're decimal creatures, lets use the more convenient interpretation of 'megabit' as 1x10^6 bits. This means it will take 18.125x10^6 "Fetch" commands to move all the data from your game from the memory to the CPU.  In the case of no compression, it will take another 18.125x10^6 'write' commands to save it to the disk.  This gives us a grand total of 36.25x10^6 actions for an uncompressed save.

We're also going to assume that the write command only works if there is a full 64 bits of data (with the remainder a pad of NULL).  It doesn't make sense to write 12.8 bits followed by 52.2 NULLs with every write, as that would end up occupying the same space on the disk as the uncompressed version.  So we're only going to write once we have 64 bits to write, and the last write will get padded with NULLs.  With a save file size of 28.5mb, we get exactly 3562500 writes (3.56x10^6).   So, assuming our algorithm above, this is 18.125x10^6 fetches, plus 18.125x10^6 compressions, plus 3.56x10^6) writes for a total of 39.8125x10^6 actions.

Above we stated the assumption that the CPU could perform 1.25x10^9 actions per second.  As a direct write takes 36.25x10^6 actions, this means an uncompressed save will take 0.029 second.  A compressed write takes 39.8125x10^6 actions, so gets done in 0.032 second.  The difference between the two is 0.003 second -- roughly 9% shorter.

 

OK, so compressing appears to be 9% slower.  However, remember that we assumed that writing to a disk takes exactly the same amount of time as reading from memory. This is almost never the case.  Lets give our hypothetical simulation the benefit of the doubt and assume that we're using slow DDR3 1600 RAM (12800 MB/s) and a fast PCI-e SSD that can write at 3000 MB/s (we're going to ignore things like sequential vs random access writes, IOPs, etc, and just use the raw posted write speed for this example).  This will end up weighting the final part of our algorithm: Writing to the disk.  Since both cases read from RAM, that's an extra 1160MB / 12800MB/s = 0.09 s for both cases. 

This means we're effectively comparing: "Write 145mb" with "Compress 145mb to 28.5mb, then write."  So lets look at the difference in time that it will take to write both files to the drive.

  • Uncompressed, 1160MB / 3000MB/s = 0.3867 s.
  • Compressed, 228MB / 3000MB/s = 0.076 s.

This gives a difference of 0.3107 s in write times between the uncompressed and the compressed file.  Since we're adding a delay to the 'write' stage, we have the following:

  • Uncompressed: Read, Write CPU time --> Delay for read (0.09 s) + CPU Time (0.029 s) + delay for write (0.3867 s) = 0.506 s.
  • Compressed: Read, Compress, Write CPU time -->  Delay for read (0.09 s) + CPU Time (0.031 s) + delay for write (0.076 s) = 0.197 s.

Including the read/write delays, we've got a difference of 0.309 s -- The uncompressed write is roughly 61% slower.  

To put this another way, 0.309 s with our assumed simulation, is 3.8x10^8 CPU "actions."  In the time it takes to write the uncompressed file, we can 'act' (get command, execute command) 21 times on each set of 64 bits of memory we're compressing and still finish writing the compressed file first (by a very small fraction of a second).  

 

***
I'm well aware that I'm ignoring things like shuffling data between registers and the like, but this example is complicated enough already.  Basically I wanted to show that if you have a reasonably fast compression algorithm (less than 21 'actions' per data set), you'll still beat writing to a very fast disk.  For most compression algorithms, you're compromising between final size and the speed to compress.  For example, it takes my computer twice as long to compress a video in H.265 than it does to actually play said video (my PC sucks).. On the other hand, that video can be 250 times smaller than the raw footage I pulled off my camera.  The OP stated their compressed file was only 5 times smaller than the uncompressed, so for this example I assumed that it was a reasonably fast compression algorithm.

Side note: I'm tired and not sure I did my percentages correctly.  I really shouldn't start a breakdown like this an hour after I should have been asleep.  Since its now nearly 1am and my alarm goes off at 5am, I'm calling it quits for the night.  

=^.^=

Share this post


Link to post
Share on other sites
JRup    164

Wonderful breakdown by @KittenIsAGeek

It does get complicated and outlining the actual subject like this gives a better perspective on what may be going on behind the curtain. I really don't want to imagine how fun calculation gets when swapping in a RAID10 drive made with said PCI-e SSDs

20 hours ago, KittenIsAGeek said:

The OP stated their compressed file was only 5 times smaller than the uncompressed

This is based on empirical observation of file sizes when only unchecking the "isCompressed" checkbox in duplicity and saving afterwards. Give or take some bytes and we ended up with some rounding that is practical :)

20 hours ago, KittenIsAGeek said:

my PC sucks

I know someone out there probably said H.265 (just checked, it's avx512, my bad ) sucks and should die a fiery death, so I probably wouldn't put the whole blame on the pc - even if it's a potato.

20 hours ago, KittenIsAGeek said:

I really shouldn't start a breakdown like this an hour after I should have been asleep.

I see you've been bitten by the worst "bug" in ONI too. The one that teleports people from 8pm to 5am (or any odd-ish "am" hour)

I raise my coffee mug to you.

Share this post


Link to post
Share on other sites
Cipupec2    47

Btw, the this whole premise is built on the assumption that isCompressed cause data to be compressed and the alternative do not manipulate data. However, what if its just different data formats e.g. if isCompressed is poorly named parm to serialize and store savefiles as binary, with the alternative taking more space but not necessarily less computing time to generate. 

Share this post


Link to post
Share on other sites
JRup    164
1 hour ago, Cipupec2 said:

this whole premise is built on the assumption that isCompressed cause data to be compressed and the alternative do not manipulate data.

Correct, I don't think devs would be able to provide more direct insight on account that even I think it's part of their secret sauce. But, hey, the code is there in duplicity for all to see too. (ATM: Too lazy to read code & track down specifics... it is indeed zlib related, go figure)
Of course, if they went and said: "hey, any guinea pigs in the room? here's the config variable that goes in this file, knock yourselves out - don't blame us" Then there would indeed be a legion of lemmings - me included, for a while at least.

Share this post


Link to post
Share on other sites
Mantak    63

This small mod turn off compression for your savegame.

 

And after you realize that turning off compression for savegame will not speed your game  saving, try this video. It will not help, but it will cheer you up :

 

 

Ok, time for some good Game (Factorio).

 

OniUncompress.dll

Share this post


Link to post
Share on other sites
JRup    164

So, mystery dll and I have to take the game apart for testing. Hmm, no krupnik for you. (I mean, I can try but then I'd have to go ahead and get into manually modding the game. Dropping in the dll just won't work.)

Found this, see if it takes anybody down memory lane :D
 

image.thumb.png.71f4c9d9e2ed67895fcd39427950476c.png

Share this post


Link to post
Share on other sites
JRup    164

Sooooo .... Had to carefully read how to install mods in linux. From the wise words by @miauly:

  • On Linux, the folder should be Dev, not dev, so: mods\Dev\SomeMod\SomeMod.dll .

(Basic mod tutorial)

Thank you @Mantak for the surprise mod.

Approximate results using a stopwatch, mileage may vary:

  • with compression: 28.6MB 20-22s
  • disabling compression: 154.3MB 13-15s

So the gain is marginal, but it's a gain nonetheless.

@Mantak gets chocochip cookie whether he likes it or not. Or shove vole meat, which ever arrives first.

 

Can this be all improved? Probably.  We could float ideas such as requesting a better compression algorithm, getting it done in the background, etc. It's still somebody's dev time that would have to go into it.
I for one would like to thank all that have chimed in and offered their insights.

I'll get to building some weird contraption later on and maybe share so we can all laugh at it together.

Share this post


Link to post
Share on other sites
SharraShimada    914

There is some possible gain. KLEI could implement a GPU-compression-routine, if a compatible GPU is present. That way the CPU would be relieved, and supported compression methods are much faster on a heavy parallelized system like a GPU. 

The downside is, not every graphics card would support ist, so its not a general solution, and you would have a fallback for everyone else.

Keep in mind, games are designed with general purpose in mind. It has to run on every system. When you start to add techniques limited to certain hardware, you, the developer, get hatestorms from those, who suffer from bad performance, due to older hardware. 

The second problem with a grown system, running for years is, that you dont just change something casually. You will break something nearly every time in the process. Remember the last Patch, that implemented the new Unity-version, changing the rand-algorithm in the process? It caused to break the whole map-generation, making all previously generated seeds obsolete and void. The worst case would be, you change something, and a vast amount of your player-base sits there, unable to play anymore.

Share this post


Link to post
Share on other sites
JRup    164
39 minutes ago, SharraShimada said:

Keep in mind, games are designed with general purpose in mind. It has to run on every system.

Correct, that's why I went as far as saying "better compression algorithm". My best guess is they're aiming for the "good enough" sweetspot.

41 minutes ago, SharraShimada said:

When you start to add techniques limited to certain hardware...

I definitely tried to veer away from exotic solutions, just in case of limiting the range of the player base as you said. The current compression algorithm does not support multicore, for example. Single core computers are but a thing of the past nowadays.

That's where I guess some mild improvement can be made. I'm still for any kind of tech that gets the thing done ok and isn't a game breaker. Still, CPUs are multicore, don't forget that guys. There's no problem with keeping the compression at CPU level but we gotta pump up those performance numbers! :D

 

 

Share this post


Link to post
Share on other sites
Gurgel    2165
20 hours ago, JRup said:

So the gain is marginal, but it's a gain nonetheless.

Fundamental engineering wisdom: Do not mess with things for a marginal gain.

Share this post


Link to post
Share on other sites
Cipupec2    47
21 hours ago, JRup said:

Can this be all improved? Probably.  We could float ideas such as requesting a better compression algorithm, getting it done in the background, etc. It's still somebody's dev time that would have to go into it.

ONI is past production, so any changes to how the engine works are unlikely, that assuming that they are actual viable and there robust multi-platform package to implement it. But if we are dreaming, then they could also use GPU for things like pathfinding optimization  

Share this post


Link to post
Share on other sites