Save game has station usage logs. Why?

[kevork]

I had a look through a late game save file with Duplicity and found that the game logs every use of at least the research station. In this particular game there where almost 5000 timestamped entries for this one station (I had dirt shortages so lots of small limited research jobs). I have not checked other stations such as metal refineries etc, but removing this logging may help reduce the huge save lags and memory requirements in later game cycles.

[Xuhybrid]

I bet it also saves all the colony stats from every cycle too.

[DarkMoge]

Well, I think that you do not realize how little does it actually take for a program to write some sort of log. I am pretty sure that if the game devs remove all the logging from the game, it will not change game performance. So, the question becomes why not? Saving more logs helps devs to detect the cause of a bug. What it does impact is the size of the save file.

As someone who could be considered a programmer, I want to say, that programmers add and should add logging pretty much everywhere, because it make debugging easier.

[kevork]

1 hour ago, DarkMoge said:

Well, I think that you do not realize how little does it actually take for a program to write some sort of log. I am pretty sure that if the game devs remove all the logging from the game, it will not change game performance. So, the question becomes why not? Saving more logs helps devs to detect the cause of a bug. What it does impact is the size of the save file.

As someone who could be considered a programmer, I want to say, that programmers add and should add logging pretty much everywhere, because it make debugging easier.

I am also a programmer of 30+ years. I agree that logging is very useful in finding bugs, however what is being generated & kept in memory and written to save files is IMHO very excessive. It takes CPU time to serialize the data into the save file, in later game cycles the save lag becomes increasingly more noticeable.

It also has entries for dupes using almost everything, door access and farm tiles are a few I checked.

Here is an example for just one door. I estimate about 60 doors in this base.

Here is the start....

And the end..  

At roughly 4 lines per entry thats 17000 entries for a single door.

Duplicity link...

https://robophred.github.io/oni-duplicity/#/raw

[Xuhybrid]

Shouldn't logs for debugging purposes be something i opt into instead of something everyone is subjected to? Especially if it significantly detracts from the play experience by pausing your game for 10 second every cycle.

[Derringer]

It almost seems like they tried to hold onto enough information to be able to replay a save from the beginning. The game engine is far too nondeterministic to use the much smaller player command logs for replay.

[kevork]

50 minutes ago, Derringer said:

It almost seems like they tried to hold onto enough information to be able to replay a save from the beginning. The game engine is far too nondeterministic to use the much smaller player command logs for replay.

Your most likely right on this.  That saved game did have replay enabled at a lowish resolution (now turned off). Klei might be using the timestamps on each item to generate the replay... 

I'm only 150 data packs away from being able to get my first gassy moo, So I'm not too keen on starting up a new game with replays disabled.

[Oxygenbreather]

Oh maybe this is why the game gets eating up more and more RAM in later cycles?

Can I remove this from my save game?

[kapoff]

30 minutes ago, Oxygenbreather said:

Oh maybe this is why the game gets eating up more and more RAM in later cycles?

Can I remove this from my save game?

Very interesting post !
Did you test that ?

[bzgzd]

4 hours ago, kevork said:

Your most likely right on this.  That saved game did have replay enabled at a lowish resolution (now turned off). Klei might be using the timestamps on each item to generate the replay...

I really doubt that they could even try to simulate whole game cycle within few seconds because of replays...

I think replay feature is just series of png images saved in directory RetiredColonies.
First that directory was inside save_files dir and about 3 weeks ago they changed it to get more stable zoom and view and also moved RetiredColonies to same dir level as save_files.

Also I checked the code and this feature (saving active/inactive operational times) was added recently in version 364722.

Probably something for more detailed graph statistics but a bit overkill?

Before it had specifically set [SkipSaveFileSerialization] flag but probably saving is quick enough so why not to add more data...

 

Spoiler

 

 

[Oxygenbreather]

1 hour ago, bzgzd said:

but probably saving is quick enough so why not to add more data...

Because it's show this data is also kept in memory, and I hardly can play anymore, because my game keeps crashing because I'm hitting my system RAM limit.

Maybe next thing I could try is to rebuild all heaviy used stations... maybe they do eat up considerable amount of RAM for this nonsense.

[Xuhybrid]

I will note that my old 2200 cycle save file is below half the size of my latest 800 cycle save file.

[bzgzd]

2 hours ago, Oxygenbreather said:

3 hours ago, bzgzd said:

but probably saving is quick enough so why not to add more data...

Because it's show this data is also kept in memory, and I hardly can play anymore, because my game keeps crashing because I'm hitting my system RAM limit.

That was a sarcasm, maybe I should use "quick" in quotes...

 

[bzgzd]

I think in 364722 (about a week ago) was added this new "Uptime" information under Properties.

That might be the reason for that save file size increase.

However it should not be kept in save file backwards for more then 10 cycles and only for buildings where it makes sense.

 

[Nutter]

This.  The #1 thing I've wanted from Klei for ages now....... the ability to turn off logging.  I feel like I spend more time waiting for the game to autosave after each cycle then I do actually playing.... that's not good.

Make it an option!  "Check here to turn off all reports"

I've never read the reports.  I'm never going to read the reports.  I'm not interested and wasting so much of my time/RAM/HD space for something that I'm never going to be even a little bit interested in is infuriating.  Please.  Give me an option to turn it all off.

[Derringer]

5 hours ago, bzgzd said:

I think in 364722 (about a week ago) was added this new "Uptime" information under Properties.

That might be the reason for that save file size increase.

I think you found the real use case. I was just speculating.

[Chthonicone]

22 hours ago, kevork said:

I am also a programmer of 30+ years. I agree that logging is very useful in finding bugs, however what is being generated & kept in memory and written to save files is IMHO very excessive. It takes CPU time to serialize the data into the save file, in later game cycles the save lag becomes increasingly more noticeable.

It also has entries for dupes using almost everything, door access and farm tiles are a few I checked.

Here is an example for just one door. I estimate about 60 doors in this base.

Here is the start....

And the end..  

At roughly 4 lines per entry thats 17000 entries for a single door.

Duplicity link...

https://robophred.github.io/oni-duplicity/#/raw

I agree that writes to the disk take a while, but you as a programmer should know that "4 lines per entry" just means 4 additional characters in the code. Specifically the '\n'. Disk writes take a while, but we shouldn't see performance problems unless the file size crosses the disk block size.

For everyone else though, writing a file to a disk can be millions of times slower than writing to memory. Maybe tens of thousands time slower instead if you use an SSD. This though shouldn't affect performance if they spliced it off into it's own thread though and maybe they should look into that.

8 hours ago, bzgzd said:

I think in 364722 (about a week ago) was added this new "Uptime" information under Properties.

That might be the reason for that save file size increase.

However it should not be kept in save file backwards for more then 10 cycles and only for buildings where it makes sense.

 

I would agree with you that this is why those logging entries were added, but this doesn't have to be the way it's implemented.

Instead they could keep track of only the last 10 days worth of entries, and generated it off of that. This would cap the file size at only so many entries.

[Xuhybrid]

6 hours ago, Nutter said:

This.  The #1 thing I've wanted from Klei for ages now....... the ability to turn off logging.  I feel like I spend more time waiting for the game to autosave after each cycle then I do actually playing.... that's not good.

Make it an option!  "Check here to turn off all reports"

I've never read the reports.  I'm never going to read the reports.  I'm not interested and wasting so much of my time/RAM/HD space for something that I'm never going to be even a little bit interested in is infuriating.  Please.  Give me an option to turn it all off.

Agreed. Also, store them separately to the save file so we can manually delete them.

[Xaekai]

So... I looked into this further and these logs account for a massive portion of the savegame data now. They made the gameObjects portion of the savedata so big I could no longer pretty print it as a JSON textfile for manual editing because the text serialization became greater then 1.8GB.

Peter Han on the Discord whipped up a quick mod that guts these logs. It reduced my save lag from 17 seconds to 11 seconds. The game now loads almost twice as fast. Here is the impact on the savefile size

-rw-r--r-- 1 xaekai xaekai   543532 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.png'
-rw-r--r-- 1 xaekai xaekai 23070467 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.sav'
-rw-r--r-- 1 xaekai xaekai   540629 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.png'
-rw-r--r-- 1 xaekai xaekai 11052971 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.sav'

As you can see it was literally half the save file. Peter says the mod will be public on the workshop soon.

[Chthonicone]

1 hour ago, Xaekai said:

So... I looked into this further and these logs account for a massive portion of the savegame data now. They made the gameObjects portion of the savedata so big I could no longer pretty print it as a JSON textfile for manual editing because the text serialization became greater then 1.8GB.

Peter Han on the Discord whipped up a quick mod that guts these logs. It reduced my save lag from 17 seconds to 11 seconds. The game now loads almost twice as fast. Here is the impact on the savefile size

-rw-r--r-- 1 xaekai xaekai   543532 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.png'
-rw-r--r-- 1 xaekai xaekai 23070467 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.sav'
-rw-r--r-- 1 xaekai xaekai   540629 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.png'
-rw-r--r-- 1 xaekai xaekai 11052971 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.sav'

As you can see it was literally half the save file. Peter says the mod will be public on the workshop soon.

Good work, did you remove all logs over 6000 seconds old?

[Xaekai]

No, the mod he made gutted all of them. Because they serve no observable function. I'm sure it would be trivial to make it preserve some.

Also, these logs are not how "Uptime" is tracked.

[Chthonicone]

1 minute ago, Xaekai said:

No, the mod he made gutted all of them. Because they serve no observable function. I'm sure it would be trivial to make it preserve some.

Also, these logs are not how "Uptime" is tracked.

Ok, good to know.

[Xuhybrid]

2 hours ago, Xaekai said:

So... I looked into this further and these logs account for a massive portion of the savegame data now. They made the gameObjects portion of the savedata so big I could no longer pretty print it as a JSON textfile for manual editing because the text serialization became greater then 1.8GB.

Peter Han on the Discord whipped up a quick mod that guts these logs. It reduced my save lag from 17 seconds to 11 seconds. The game now loads almost twice as fast. Here is the impact on the savefile size

-rw-r--r-- 1 xaekai xaekai   543532 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.png'
-rw-r--r-- 1 xaekai xaekai 23070467 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.sav'
-rw-r--r-- 1 xaekai xaekai   540629 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.png'
-rw-r--r-- 1 xaekai xaekai 11052971 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.sav'

As you can see it was literally half the save file. Peter says the mod will be public on the workshop soon.

Knew it! I'd say it's worth keeping like 10 cycles default. Even if i never use them.

[Lifegrow]

On 15/09/2019 at 9:46 AM, Xaekai said:

No, the mod he made gutted all of them. Because they serve no observable function. I'm sure it would be trivial to make it preserve some.

Also, these logs are not how "Uptime" is tracked.

Any news on this mod being put on Steam Workshop ?

[bobucles]

My question is why is time expressed as a float when simulation ticks are very clearly discrete values?