kevork

Save game has station usage logs. Why?

Recommended Posts

kevork    23

I had a look through a late game save file with Duplicity and found that the game logs every use of at least the research station. In this particular game there where almost 5000 timestamped entries for this one station (I had dirt shortages so lots of small limited research jobs). I have not checked other stations such as metal refineries etc, but removing this logging may help reduce the huge save lags and memory requirements in later game cycles.

  • Like 2
  • Thanks 1

Share this post


Link to post
Share on other sites
DarkMoge    138

Well, I think that you do not realize how little does it actually take for a program to write some sort of log. I am pretty sure that if the game devs remove all the logging from the game, it will not change game performance. So, the question becomes why not? Saving more logs helps devs to detect the cause of a bug. What it does impact is the size of the save file.

As someone who could be considered a programmer, I want to say, that programmers add and should add logging pretty much everywhere, because it make debugging easier.

Edited by DarkMoge
  • Like 1

Share this post


Link to post
Share on other sites
Xuhybrid    259

Shouldn't logs for debugging purposes be something i opt into instead of something everyone is subjected to? Especially if it significantly detracts from the play experience by pausing your game for 10 second every cycle.

  • Like 2

Share this post


Link to post
Share on other sites
Derringer    25

It almost seems like they tried to hold onto enough information to be able to replay a save from the beginning. The game engine is far too nondeterministic to use the much smaller player command logs for replay.

Share this post


Link to post
Share on other sites
kevork    23
50 minutes ago, Derringer said:

It almost seems like they tried to hold onto enough information to be able to replay a save from the beginning. The game engine is far too nondeterministic to use the much smaller player command logs for replay.

Your most likely right on this.  That saved game did have replay enabled at a lowish resolution (now turned off). Klei might be using the timestamps on each item to generate the replay... 

I'm only 150 data packs away from being able to get my first gassy moo, So I'm not too keen on starting up a new game with replays disabled.

Share this post


Link to post
Share on other sites
kapoff    2
30 minutes ago, Oxygenbreather said:

Oh maybe this is why the game gets eating up more and more RAM in later cycles?

Can I remove this from my save game?

Very interesting post !
Did you test that ?

Share this post


Link to post
Share on other sites
bzgzd    272
4 hours ago, kevork said:

Your most likely right on this.  That saved game did have replay enabled at a lowish resolution (now turned off). Klei might be using the timestamps on each item to generate the replay...

I really doubt that they could even try to simulate whole game cycle within few seconds because of replays...

I think replay feature is just series of png images saved in directory RetiredColonies.
First that directory was inside save_files dir and about 3 weeks ago they changed it to get more stable zoom and view and also moved RetiredColonies to same dir level as save_files.


Also I checked the code and this feature (saving active/inactive operational times) was added recently in version 364722.

Probably something for more detailed graph statistics but a bit overkill?

Before it had specifically set [SkipSaveFileSerialization] flag but probably saving is quick enough so why not to add more data... :)

 

Spoiler

operational_diff.thumb.png.38d1d49298c679d5cf5465dbe829ef19.png

 

 

  • Like 1
  • Haha 1

Share this post


Link to post
Share on other sites
1 hour ago, bzgzd said:

but probably saving is quick enough so why not to add more data... :)

Because it's show this data is also kept in memory, and I hardly can play anymore, because my game keeps crashing because I'm hitting my system RAM limit.

Maybe next thing I could try is to rebuild all heaviy used stations... maybe they do eat up considerable amount of RAM for this nonsense.

Share this post


Link to post
Share on other sites
Xuhybrid    259

I will note that my old 2200 cycle save file is below half the size of my latest 800 cycle save file.

Share this post


Link to post
Share on other sites
bzgzd    272
2 hours ago, Oxygenbreather said:
3 hours ago, bzgzd said:

but probably saving is quick enough so why not to add more data... :)

Because it's show this data is also kept in memory, and I hardly can play anymore, because my game keeps crashing because I'm hitting my system RAM limit.

That was a sarcasm, maybe I should use "quick" in quotes...

 

Share this post


Link to post
Share on other sites
bzgzd    272

I think in 364722 (about a week ago) was added this new "Uptime" information under Properties.

That might be the reason for that save file size increase.

However it should not be kept in save file backwards for more then 10 cycles and only for buildings where it makes sense.

uptime.png.6d137f2efd3178acbab3192bc893d413.png
 

  • Like 1
  • Thanks 1

Share this post


Link to post
Share on other sites
Nutter    24

This.  The #1 thing I've wanted from Klei for ages now....... the ability to turn off logging.  I feel like I spend more time waiting for the game to autosave after each cycle then I do actually playing.... that's not good.

Make it an option!  "Check here to turn off all reports"

I've never read the reports.  I'm never going to read the reports.  I'm not interested and wasting so much of my time/RAM/HD space for something that I'm never going to be even a little bit interested in is infuriating.  Please.  Give me an option to turn it all off.

  • Like 2

Share this post


Link to post
Share on other sites
Derringer    25
5 hours ago, bzgzd said:

I think in 364722 (about a week ago) was added this new "Uptime" information under Properties.

That might be the reason for that save file size increase.

I think you found the real use case. I was just speculating.

Share this post


Link to post
Share on other sites
Chthonicone    49
22 hours ago, kevork said:

I am also a programmer of 30+ years. I agree that logging is very useful in finding bugs, however what is being generated & kept in memory and written to save files is IMHO very excessive. It takes CPU time to serialize the data into the save file, in later game cycles the save lag becomes increasingly more noticeable.

It also has entries for dupes using almost everything, door access and farm tiles are a few I checked.

Here is an example for just one door. I estimate about 60 doors in this base.

Here is the start....

image.png.7d19c2a2a53b963368d9bc6e45c5467b.png

And the end..  

image.thumb.png.e596d82b3bb0dc05c1de0810c23504db.png

At roughly 4 lines per entry thats 17000 entries for a single door.

Duplicity link...

https://robophred.github.io/oni-duplicity/#/raw

I agree that writes to the disk take a while, but you as a programmer should know that "4 lines per entry" just means 4 additional characters in the code. Specifically the '\n'. Disk writes take a while, but we shouldn't see performance problems unless the file size crosses the disk block size.

For everyone else though, writing a file to a disk can be millions of times slower than writing to memory. Maybe tens of thousands time slower instead if you use an SSD. This though shouldn't affect performance if they spliced it off into it's own thread though and maybe they should look into that.

8 hours ago, bzgzd said:

I think in 364722 (about a week ago) was added this new "Uptime" information under Properties.

That might be the reason for that save file size increase.

However it should not be kept in save file backwards for more then 10 cycles and only for buildings where it makes sense.

uptime.png.6d137f2efd3178acbab3192bc893d413.png
 

I would agree with you that this is why those logging entries were added, but this doesn't have to be the way it's implemented.

Instead they could keep track of only the last 10 days worth of entries, and generated it off of that. This would cap the file size at only so many entries.

  • Like 1
  • Confused 1

Share this post


Link to post
Share on other sites
Xuhybrid    259
6 hours ago, Nutter said:

This.  The #1 thing I've wanted from Klei for ages now....... the ability to turn off logging.  I feel like I spend more time waiting for the game to autosave after each cycle then I do actually playing.... that's not good.

Make it an option!  "Check here to turn off all reports"

I've never read the reports.  I'm never going to read the reports.  I'm not interested and wasting so much of my time/RAM/HD space for something that I'm never going to be even a little bit interested in is infuriating.  Please.  Give me an option to turn it all off.

Agreed. Also, store them separately to the save file so we can manually delete them.

  • Like 1

Share this post


Link to post
Share on other sites
Xaekai    20

So... I looked into this further and these logs account for a massive portion of the savegame data now. They made the gameObjects portion of the savedata so big I could no longer pretty print it as a JSON textfile for manual editing because the text serialization became greater then 1.8GB.

Peter Han on the Discord whipped up a quick mod that guts these logs. It reduced my save lag from 17 seconds to 11 seconds. The game now loads almost twice as fast. Here is the impact on the savefile size

-rw-r--r-- 1 xaekai xaekai   543532 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.png'
-rw-r--r-- 1 xaekai xaekai 23070467 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.sav'
-rw-r--r-- 1 xaekai xaekai   540629 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.png'
-rw-r--r-- 1 xaekai xaekai 11052971 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.sav'

As you can see it was literally half the save file. Peter says the mod will be public on the workshop soon.

  • Like 2
  • Thanks 2

Share this post


Link to post
Share on other sites
Chthonicone    49
1 hour ago, Xaekai said:

So... I looked into this further and these logs account for a massive portion of the savegame data now. They made the gameObjects portion of the savedata so big I could no longer pretty print it as a JSON textfile for manual editing because the text serialization became greater then 1.8GB.

Peter Han on the Discord whipped up a quick mod that guts these logs. It reduced my save lag from 17 seconds to 11 seconds. The game now loads almost twice as fast. Here is the impact on the savefile size


-rw-r--r-- 1 xaekai xaekai   543532 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.png'
-rw-r--r-- 1 xaekai xaekai 23070467 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.sav'
-rw-r--r-- 1 xaekai xaekai   540629 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.png'
-rw-r--r-- 1 xaekai xaekai 11052971 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.sav'

As you can see it was literally half the save file. Peter says the mod will be public on the workshop soon.

Good work, did you remove all logs over 6000 seconds old?

Share this post


Link to post
Share on other sites
Xaekai    20

No, the mod he made gutted all of them. Because they serve no observable function. I'm sure it would be trivial to make it preserve some.

Also, these logs are not how "Uptime" is tracked.

Share this post


Link to post
Share on other sites
Chthonicone    49
1 minute ago, Xaekai said:

No, the mod he made gutted all of them. Because they serve no observable function. I'm sure it would be trivial to make it preserve some.

Also, these logs are not how "Uptime" is tracked.

Ok, good to know.

Share this post


Link to post
Share on other sites
Xuhybrid    259
2 hours ago, Xaekai said:

So... I looked into this further and these logs account for a massive portion of the savegame data now. They made the gameObjects portion of the savedata so big I could no longer pretty print it as a JSON textfile for manual editing because the text serialization became greater then 1.8GB.

Peter Han on the Discord whipped up a quick mod that guts these logs. It reduced my save lag from 17 seconds to 11 seconds. The game now loads almost twice as fast. Here is the impact on the savefile size


-rw-r--r-- 1 xaekai xaekai   543532 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.png'
-rw-r--r-- 1 xaekai xaekai 23070467 2019-09-14 22:38 'Hydrocarbon Eden Cycle 1130.sav'
-rw-r--r-- 1 xaekai xaekai   540629 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.png'
-rw-r--r-- 1 xaekai xaekai 11052971 2019-09-14 23:46 'Hydrocarbon Eden Cycle 1131.sav'

As you can see it was literally half the save file. Peter says the mod will be public on the workshop soon.

Knew it! I'd say it's worth keeping like 10 cycles default. Even if i never use them.

Share this post


Link to post
Share on other sites
Lifegrow    1,496
On 15/09/2019 at 9:46 AM, Xaekai said:

No, the mod he made gutted all of them. Because they serve no observable function. I'm sure it would be trivial to make it preserve some.

Also, these logs are not how "Uptime" is tracked.

Any news on this mod being put on Steam Workshop ?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now