Memory HOG? Game Lags late game

[RonEmpire]

So late game,  I have a feeling that the memory that is bloating because of the COMPOUNDING cycle reports that is being kept in memory (and growing) is the long term cause of lag.

What's the cap limit to memory?  Eventually you will run out of memory and even with virtual memory page swapping its going to boggle your computer down. 

There's only so many calculations you can do per tile (per elements).   But the real problem is the growing memory hog.

 

[Gurgel]

Well, it may be the cycle reports, but only if they are somehow stored in raw form. In the form displayed, they can hardly be more than 1kB or so per report and that would not cause memory problems.

[Etiam]

 @RonEmpire It seems like all cycle reports take much space in save file - just as we spoke in this thread:

And they probably can take some memory in game, but I think it the real problem are memory leaks.

@Gurgel Sadly these reports seem to be stored in the "raw" form, whole collection always in memory.

After I removed the daily reports using the mod I made "RemoveDailyReports" (link in signature below) there was major reduction of saving time, loading time (~20% reduction) and save file size (biggest was 20mb to 14mb save file size decrease) for late-game worlds. Also saves are compressed so it has to be a lot more than 1kb per report

Not sure about the actual memory consumption as its kind of hard to measure.

[Gurgel]

6 hours ago, Etiam said:

And they probably can take some memory in game, but I think it the real problem are memory leaks.

@Gurgel Sadly these reports seem to be stored in the "raw" form, whole collection always in memory.

Well, obviously memory leaks, but if the reports take this much data, there should be an option to limit the history stored. I almost never use the reports anyways.

[LeftyRighty]

unless you're using "memory leak" colloquially... those aren't symptoms of memory leaks; if it was a memory leak closing the game and reloading from the same spot would improve performance. 

[Etiam]

@LeftyRighty And it does, you can simply very easily:

1. Save the game.
2. Load that save.
3. Look how game eats up more memory.
4. Go to 2.

When I was talking about memory leak it wasn't about the reports at all, but I'm sure they cripple saving/loading time and save file size.

Skunky's save (at nearly 3000 cycles) reduced from 20mb to 14mb, loading time 24s to 19s and saving time 8.5s to 6s after removing reports.

@Gurgel So just use my mod "RemoveDailyReports" from my signature ;d it removes all but the most recent reports (currently spares last 5)

[Copy3ms]

39 minutes ago, Etiam said:

so just use my mod "RemoveDailyReports" from my signature ;d it removes all but the most recent reports (currently spares last 5)

can this help the memory issue? i honestly cant go beyond 400 cycles without eating up all of my 8gb of ram. 

Edit: cant seem to find this mod. If you happen to have a link for it. I'd appreciate it. 

[Gurgel]

5 hours ago, LeftyRighty said:

unless you're using "memory leak" colloquially... those aren't symptoms of memory leaks; if it was a memory leak closing the game and reloading from the same spot would improve performance. 

Not necessarily. If the data-structures saved on disk contain the leaked memory, memory-leaks can become persistent. 

[Lawnmower Man]

8 minutes ago, Copy3ms said:

can this help the memory issue? i honestly cant go beyond 400 cycles without eating up all of my 8gb of ram. 

Edit: cant seem to find this mod. If you happen to have a link for it. I'd appreciate it. 

I'm almost cycle 700, and I'm only using about 6 GB of RAM.  However, I do have 32 GB total and 16 GB in use by the OS, but most of it must be caches, because the next biggest consumer is Chrome, at 2 GB.  Also, you might have a lot more memory-intensive buildings on your map than I do.

6 minutes ago, Gurgel said:

Not necessarily. If the data-structures saved on disk contain the leaked memory, memory-leaks can become persistent. 

If the program can access the memory to write it to disk, let alone *reload it*, then by definition it is not a "leak".  Maybe "waste" on the part of the programmers, but definitely not a "leak".

[Copy3ms]

Just now, Lawnmower Man said:

I'm almost cycle 700, and I'm only using about 6 GB of RAM.  However, I do have 32 GB total and 16 GB in use by the OS, but most of it must be caches, because the next biggest consumer is Chrome, at 2 GB.  Also, you might have a lot more memory-intensive buildings on your map than I do.

When i play ON i have EVERYTHING closed. Trust me. I dont even have a browser open (also i dont use Chrome). I am still experiencing problems. 

[Lawnmower Man]

1 hour ago, Copy3ms said:

When i play ON i have EVERYTHING closed. Trust me. I dont even have a browser open (also i dont use Chrome). I am still experiencing problems. 

Well, I did just buy a new gaming laptop, so the CPU is also pretty new.  Since the fluid engine is now multi-threaded, it's possible it does weird things on CPUs that don't have enough cycles for it.  Parallelism creates lots of hard-to-reproduce behaviors, especially across a wide range of hardware.  Might be time to save up for a new machine. :/

[Etiam]

@Copy3ms

4 hours ago, Copy3ms said:

can this help the memory issue? i honestly cant go beyond 400 cycles without eating up all of my 8gb of ram.

Not likely, removing reports frees up only about 50mb in memory, but its significant when it comes to saving it.

4 hours ago, Copy3ms said:

Edit: cant seem to find this mod. If you happen to have a link for it. I'd appreciate it. 

Sorry, it seems like signatures are not visible by default (thanks for heads up @Cairath), heres the thread:

 

[Gurgel]

3 hours ago, Lawnmower Man said:

If the program can access the memory to write it to disk, let alone *reload it*, then by definition it is not a "leak".  Maybe "waste" on the part of the programmers, but definitely not a "leak".

Memory leak is all memory that is not needed anymore but is not freed by the application. You are thinking about "lost" memory that the application cannot free anymore (or the GC cannot identify as unused), but that is just one form of memory leakage. These usually are only persistent if the process memory map or specific parts of it are stored on disk instead of crawling the data-structures for creating save game files.

Typically, talking about a memory leak also implies it is a dynamic leak, i.e. the problem gets worse with time. There are static leaks also, that do not get worse, but just waste a fixed amount of memory.

Of course, a lot of CS terminology is fluid. But this is the mainstream usage: https://en.wikipedia.org/wiki/Memory_leak

[Lawnmower Man]

37 minutes ago, Gurgel said:

Memory leak is all memory that is not needed anymore but is not freed by the application. You are thinking about "lost" memory that the application cannot free anymore (or the GC cannot identify as unused), but that is just one form of memory leakage. These usually are only persistent if the process memory map or specific parts of it are stored on disk instead of crawling the data-structures for creating save game files.

Typically, talking about a memory leak also implies it is a dynamic leak, i.e. the problem gets worse with time. There are static leaks also, that do not get worse, but just waste a fixed amount of memory.

Of course, a lot of CS terminology is fluid. But this is the mainstream usage: https://en.wikipedia.org/wiki/Memory_leak

If you run a leak detector like valgrind or purify, you will not get "semantic leaks" where the programmer could have released a resource but didn't.  You will only get *actual* memory leaks, where the memory manager no longer has access to a block of memory.  If I ask you to write a leak detector using your definition, you will either say it's impossible, or it will just point at the entire program heap and say: "possible leak detected."

Also, for you to point at anything saved on disk and say: "That's a leak" is extremely presumptuous.  You would have to know how all the memory is not only actually used, but how it is intended to be used, as well as how it may be used in the near future.  In short, it's making a judgment on the entire codebase without actually seeing the code.  At best, we can say that the program may not be using memory as efficiently as it could.  But even then, engineering is all about making trade-offs.  And sometimes you make what looks like a bad trade-off today because you are planning something bigger tomorrow.

Since part of the engine is written in C++, according to some threads, there may be bona fide memory leaks in the code.  But I wouldn't assume that unless I ran a memory profiler on it myself and could inspect the code to see how each block is used, since profilers don't know how long-lived caches are used.

The point being, just because the memory usage grows over the lifetime of a game does not mean that the usage is not entirely by design.  It may simply be a necessary consequence of caching more complex aspects of the game, and a legitimate design decision to make the game runnable with lots of things going on.

[Gurgel]

16 hours ago, Lawnmower Man said:

If you run a leak detector like valgrind or purify, you will not get "semantic leaks" where the programmer could have released a resource but didn't. 

The thing is that Valgrind and purify cannot identify all leaks. They can only identify lost memory, not general memory leakage. This is not surprising and probably eventually goes down to the halting-problem. Your argument that only leaks that can be detected algorithmically are leaks is flawed.

16 hours ago, Lawnmower Man said:

Also, for you to point at anything saved on disk and say: "That's a leak" is extremely presumptuous. 

First, I did not actually do that. I did merely point out that memory leaks can go to saved state and be persistent that way and that a save/reload not reducing memory is not a reliable indicator for the absence of a memory-leak. Are you disputing that?  Hence I did not state that this is definitely a memory leak (I do not know that), I did only state that the save/reload test is not conclusive.

As to being "presumptuous", that depends on my qualification and experience level, now does it? Let's not go into that, because that is something you cannot win and hence I will not even start that comparison. Lets stay on facts, shall we? And that Wikipedia article I linked says "a computer program incorrectly manages memory allocations in such a way that memory which is no longer needed is not released".  Do you claim this is not the mainstream usage of the term?
 

[Lawnmower Man]

3 hours ago, Gurgel said:

[...]

And that Wikipedia article I linked says "a computer program incorrectly manages memory allocations in such a way that memory which is no longer needed is not released".  Do you claim this is not the mainstream usage of the term?

That depends on who you are talking to.  Software engineers are very sloppy with terminology, because they are usually asked to build things quickly at the expense of breaking things or introducing bugs.  Computer scientists, on the other hand, do not have the luxury of tossing around terms like "no longer needed", because they are not precise and rely on fuzzy concepts like programmer intent.  This is why garbage collectors must use formal notions like reachability from root nodes.

If a block of memory is not used for a long time, there can be several reasons:

1. All references to the block have been lost.  A "hard leak".
2. References exist, but the program will never use the block again, for all inputs and valid state transitions of the program.
3. The block is "live" by any reasonable definition, but the program will not appear to use it on the code paths which are most visible to the user.
4. The block is still useful, but just hasn't been used recently.

I'm willing to include 2 in the category of "leaks", because if a GC could detect this condition, it would be correct for it to collect the block, even though the condition is, in general, undecidable.  This would cover scenarios like cycles in a ref-counted collector.

However, writing a block to disk and loading it again most certainly doesn't qualify for 2.  It must be in 3 or 4.  I think you are trying to argue that 3 includes things we should call "memory leaks", and that the Wikipedia definition supports that.  I have lots of terms to describe code which produces 3, but "memory leak" isn't one of them.

After all, you can read "correctness" to mean the formal notion, where the program *cannot and will not* use the block in the future, which I think corresponds to 1 and 2, or you can read it as a fuzzy "program probably won't use this again, but I haven't analyzed all the code paths to be sure."  At best, I would call the latter "sloppy coding".

For instance, let's take translated strings.  If the game loads all translations, but I run it with only en_US, does the fact that zh_CN was never used before I shut it down mean that all the Chinese strings were a memory leak?  1 and 2 say "no", definitively, but you could argue either way for 3.  That's my problem with the fuzzy definition.  You could argue that a super-efficient program would only load the localizations that are actually requested by the user.  But if all the strings fit in a few MB, I would hardly fault the devs for just loading them all greedily so they don't have to touch the disk later if you decide to switch langs.

[Gurgel]

14 hours ago, Lawnmower Man said:

That depends on who you are talking to. 

I agree on that. This here is not a CS forum, so "sloppy" it is. In "sloppy", a memory leak is memory being consumed and staying consumed without actual use or need and causing problems by that. Note that most CS types (I am one) will also use the "sloppy" version when talking about the real-world issue. Purely theoretical CS types may be a different case, but you do not talk to them about real-world engineering problems anyways as they have very little to contribute there. And not even purely theoretical CS types have a hard, definite, universal definition of "memory leak", unless they go to theoretical machine models like the Turing Machine.

But you can still have leaks in your class 1 go to disk. It depends on the level you work on and something always has a reference to any piece of memory (at the very least the OS has). Example: Say you use an engine that does its own non-GC memory management. Hence there will always be references to all memory, but your application-code may not have any links to some of that memory anymore. But then this example-engine may also offer a "save/load" function that dumps and loads everything ever allocated and not explicitly freed. It cannot know that the application has no link to some block anymore and will hence store and load memory leaked on application layer. For the application, this is a hard leak, For the engine this is a "soft" leak. For the OS, this is not a leak at all. 

How why would anybody do such a "save/load" function? Simple: Because it is much, much faster than crawling the data-structures and it is generic, i.e. you do not need to know what is actually in there. This assumes you can place the memory at the same address after loading or that all access uses some kind of base-pointer. 

The thing is however, that you cannot even identify all hard leaks automatically and may misidentify them as well (the application may do pointer arithmetic, for example), so distinguishing them absolutely from "soft" leaks generally makes no sense when talking about real-world issues. This is also the reason that there is no general GC for C. For example, the Boehm–Demers–Weiser garbage collector requires that some things are not to be done on your C code. The BDW-GC also may not collect unused memory because it misinterprets bit-patterns as pointers to memory when they are actually something else.

In the end, the issue is complicated (as so often). But I would maintain that for the purpose of using it there, the general definition from wikipedia is entirely fine. 

[Copy3ms]

On 1/21/2019 at 3:14 PM, Etiam said:

@Copy3ms

Not likely, removing reports frees up only about 50mb in memory, but its significant when it comes to saving it.

So, i used parts of my home pc to beef up the PC at work, 16GB of ram now, R9280x and AMD FX-8150 @3.6 still lagging on the 400+ cycles mark. I am not even sure what to do anymore. 

It's literately unplayable for me at this stage. Cant exactly build stuff when frames drop to 1 every second. 

silently cries 

[The Plum Gate]

59 minutes ago, Copy3ms said:

So, i used parts of my home pc to beef up the PC at work, 16GB of ram now, R9280x and AMD FX-8150 @3.6 still lagging on the 400+ cycles mark. I am not even sure what to do anymore. 

It's literately unplayable for me at this stage. Cant exactly build stuff when frames drop to 1 every second. 

silently cries 

What kind of lag?

I get these weird tiling lags as if the camera is catching on something, and occasional tearing. very rare though the tearing that is.

I've got a Ryzen 1800x, vega 64, 16gb ddr4 3200... the game only uses around 1.9gb of vram for me. and the video card rarely exceeds 30% usage. So I really don't know what's going on. I play with radeon chill enabled, my framerate rarely drops below the value I have set according to steam. It's a bit awkward though almost as if there's a container running the simulation at an independent frame rate - this might just be how Unity works with 2d stuff, some sort of compositor system.

I usually go into task manager and set the core affinity for the game's exe file - take it off core 0,1 - I can say that the game utilizes one core at around 70%, another core at around 40% and the other 12 threads vary between 5% and 15% above baseline - that being me having all that other stuff going on...

I doubt me watching or listening to music via chrome is affecting anything, i've also got three monitors running - playing the game on a 3440x1440 panel, browsers or sensors running on the others. I haven't opened ryzen master to see if it's boosting over 3.6ghz, but the average utilization of the processor is around 40% at 3.2ghz.

I know trying to watch a video on one of the other monitors is like a deathblow for the game. It will lag like crazy with chrome trying to decode vp9 in the shaders while the game is running. I might buy a cheap video card and plug the other monitors into that...might solve that problem, but I don't know

[Copy3ms]

45 minutes ago, The Plum Gate said:

What kind of lag?

The frames jump from 1 to 14 constantly. It's like when the cycle is loading, but constantly. I tried the core affinity you mentioned. but nada. I honestly dont know what else i can do. Obviously is not the hardware on the pc, is sufficient to run it. TBH, this all started with the Rocket upgrade. Hopefully as the game gets patched this problem is solved. 

I really doubt it's you watching videos or having chrome on, I think it's just the current state of the game for a lot of us. 

[Etiam]

1 hour ago, Copy3ms said:

So, i used parts of my home pc to beef up the PC at work, 16GB of ram now, R9280x and AMD FX-8150 @3.6 still lagging on the 400+ cycles mark. I am not even sure what to do anymore. 

It's literately unplayable for me at this stage. Cant exactly build stuff when frames drop to 1 every second. 

silently cries 

It really shouldn't be that bad... You're exaggerating or doing something wrong, AMD with many weak cores isn't the best deal for this game either.

U should probably look for some tips how to improve performance, like making pathfinding easier, clearing debris, etc. Maybe u have used debug mode and thus revealed the whole map?

[Copy3ms]

21 minutes ago, Etiam said:

It really shouldn't be that bad... You're exaggerating or doing something wrong, AMD with many weak cores isn't the best deal for this game either.

U should probably look for some tips how to improve performance, like making pathfinding easier, clearing debris, etc. Maybe u have used debug mode and thus revealed the whole map?

Map: about 80% discovered. Im at the stage of the game where i am launching expeditions to space (cant play the game if you wont explore all the geysers/resources you have) 

Exaggerating: Well the person with the problem does tend to see things worse than they really are, but honestly i would take 20FPS any-day on this game. 

AMD; while not a fanboy, this processor should be more than sufficient for a game of this caliber, sure Intel could squeeze a few more frames, but having AMD shouldn't be a deterrent to play this game. 

Tips how to improve performance: isnt this the point of this forum? to find clues how to fix such problems? Also, i do keep my base tidy. 

Also, i dont even use Jet Packs because i know they tend to lag the game. Transit tubes are good enough. 

[Etiam]

5 minutes ago, Copy3ms said:

Map: about 80% discovered. Im at the stage of the game where i am launching expeditions to space (cant play the game if you wont explore all the geysers/resources you have) 

Exaggerating: Well the person with the problem does tend to see things worse than they really are, but honestly i take 20FPS any-day on this game. 

AMD; while not a fanboy, this processor should be more than sufficient for a game of this caliber, sure Intel could squeeze a few more frames, but having AMD shouldn't be a deterrent to play this game. 

Tips how to improve performance: isnt this the point of this forum? to find clues how to fix such problems? Also, i do keep my base tidy. 

Also, i dont even use Jet Packs because i know they tend to lag the game. Transit tubes are good enough. 

I'm pretty sure your CPU is the bottleneck at this point. I have a CPU that is about 50% better when it comes to single core performance and yet it still tends to dip to like 25-30fps at majority of map revealed (no rockets launched, but surface breached, I'd say about 50% of the map revealed).

You might have been deceived by the lack of 3D in this game or simple (but pretty!) graphics, when you were judging this game's "caliber" It's very CPU intensive, just consider the heat transfer... on fully revealed map you have 256*384=98 304 cells that transfer heat 5 times per second between each other, then add to it all buildings and debris lying around transfering heat too - that's a lot of calculations. And this is just heat transfer, think about gases and liquids movement, dupes pathfinding, critter movement, gas and liquid pipes, buildings and geysers spewing out new elements.

There's probably a lot of room to improve in the game itself, but it will remain CPU-intensive, so upgrade your PC or find out whats wrong with your world. I heard locking out storages (in lockers) in vacuum with no dupe access helps greatly.

Actually I think smaller worlds would help for lower-end machines.

[The Plum Gate]

I had an fx 8320 and a gtx760, rx480. Didn't help too much by increasing the frequency. You may be right regarding the state of the game for the majority of us.

[Gurgel]

1 hour ago, Etiam said:

I'm pretty sure your CPU is the bottleneck at this point.

I use an FX8350 with 16GB RAM. (Waiting for Zen-2 to upgrade.) It is only about 12% faster than the FX8150.  No problems with frame rates. I suspect something else on the system is causing the problem.