Huge RAM issue

[Oozinator]

54 minutes ago, ToiDiaeRaRIsuOy said:

I was looking for an excuse to get a Linux install, so yay!

Don't forget to report back, if success!

[ToiDiaeRaRIsuOy]

1 minute ago, Oozinator said:

Don't forget to report back, if success!

I'll double down on that. When the time comes, I'll open up a topic where I'll ask people to send me their low fps save files. I'll test them on the same system, both on Windows and Linux (Mint?) and share the results.

[bobucles]

Sometimes RAM sticks go bad. You may rarely see any consequences in normal use, but it's enough to crash a memory intensive game like ONI. I learned I had a bad memory stick only after reaching late game Factorio.

Check out memtest86 (dot ORG, not dot com). Boot up with it and let it run overnight. Any number of errors is too many and indicates some kind of physical problem with your computer. 

Unfortunately operating systems don't let you just ignore a bad pixel of memory, so 99.99% good isn't good enough. Some obscure Unix environment might allow it.

[ToiDiaeRaRIsuOy]

1 minute ago, bobucles said:

Sometimes RAM sticks go bad. You may rarely see any consequences in normal use, but it's enough to crash a memory intensive game like ONI. I learned I had a bad memory stick only after reaching late game Factorio.

Check out memtest86 (dot ORG, not dot com). Boot up with it and let it run overnight. Any number of errors is too many and indicates some kind of physical problem with your computer. 

Unfortunately operating systems don't let you just ignore a bad pixel of memory, so 99.99% good isn't good enough. Some obscure Unix environment might allow it.

I do think given the widespread issue on RAM we aren't talking about dodgy memory in general. Of course, that can still come on top of that, and probably reducing your game completely down to a crawl.

[bobucles]

1 hour ago, ToiDiaeRaRIsuOy said:

do think given the widespread issue on RAM we aren't talking about dodgy memory in general. Of course, that can still come on top of that, and probably reducing your game completely down to a crawl.

Faulty RAM mostly causes crashes and very rarely black magic. It doesn't cause slow performance, though the solution can include slowing the RAM down. 

Memtest is still a great diagnostic tool for crossing off potential causes of crashes. When in doubt, test it out.

[ToiDiaeRaRIsuOy]

Just now, bobucles said:

Faulty RAM mostly causes crashes and very rarely black magic. It doesn't cause slow performance, though the solution can include slowing the RAM down. 

Memtest is still a great diagnostic tool for crossing off potential causes of crashes. When in doubt, test it out.

That's a good tip actually .

[Oxygenbreather]

9 hours ago, kapoff said:

Glad to read this. Didn't knew thas Unity could be really better under Linux.

I'm not sure I could generalize that.

One thing to note, is that on my 8GB system, windows needs about 2GB just being Windows, and thats after I optimized already to kill several unnecessary processes, like search, prestart optimization, hinder steam "Webhelper" from running. etc. but it's still a hell lot of stuff thats running.

On Linux starting normally with Gnome, without any optimization, I've 7GB RAM available to be used.

It's still crashing and I had a few freezes. As I don't have another system at home, I cannot tell if just screen/keyboard went bad or the kernel, as I can't try to ssh into my machine which I usually do, if the graphic system crashes. But it crashes/freezes after about 10-20 cycles, if I don't close and restart ONI in between contrary to windows where it crashes at the latest on the second morning save.

Also this laptop has just a Intel HD chip, for this Linux drivers are as far I can tell quite good. Nvidia drivers for example, can be a hassle.

I could run memtest again, but I doubt thats an issue. I did quite a bunch of scientific computing on this machine too, never was an issue.

[kapoff]

21 hours ago, Oxygenbreather said:

I'm not sure I could generalize that.

One thing to note, is that on my 8GB system, windows needs about 2GB just being Windows, and thats after I optimized already to kill several unnecessary processes, like search, prestart optimization, hinder steam "Webhelper" from running. etc. but it's still a hell lot of stuff thats running.

On Linux starting normally with Gnome, without any optimization, I've 7GB RAM available to be used.

It's still crashing and I had a few freezes. As I don't have another system at home, I cannot tell if just screen/keyboard went bad or the kernel, as I can't try to ssh into my machine which I usually do, if the graphic system crashes. But it crashes/freezes after about 10-20 cycles, if I don't close and restart ONI in between contrary to windows where it crashes at the latest on the second morning save.

Also this laptop has just a Intel HD chip, for this Linux drivers are as far I can tell quite good. Nvidia drivers for example, can be a hassle.

I could run memtest again, but I doubt thats an issue. I did quite a bunch of scientific computing on this machine too, never was an issue.

You are right I think.
Could you share your save file ?

I specify that i don't close my many applications I never close (many firefox, thunderbird, many ssh console and other stuff) This my working station (yes I know, it is bad).
In my experience, Linux try to put in RAM everything he can. so never look at my free RAM.

Here is my save game file for compare Windows and Linux.
I will try to test windows save game also if anybody want.

LaunchUpdate.sav

[Oxygenbreather]

I ran today MemCheck86 for a few hours, 4 passes no issues (as expected)

Pandora.sav

PS: the "free" command in the console gives a realistic view of "free" memory on the last column. Other tools often "falsely" report close to zero all the time, as the kernel will just use everything there is for caching (e.g. why throw that binary image  away, if you maybe could use it later as long nothing else needs the space)

[kapoff]

41 minutes ago, Oxygenbreather said:

I ran today MemCheck86 for a few hours, 4 passes no issues (as expected)

Pandora.sav

I loaded your save game.
Press H key then waiting few sec

x1 : 55 fps
x2 : 34 fps
x3 : 25 fps

actually playing with it at max speed for test

Edit : no crash (like your volcano taming )

[Oxygenbreather]

I don't care so much about fps. And I suppose you just got more RAM than me, which IMO is the whole point of this thread. The issues start, when RAM becomes scarce.

[xialeth]

I just downloaded your save and started it up on my computer (Win10, Intel i5-4570S @2.90GHz, 8GB Ram), and so far it's run fine for five cycles. It climbed precariously high wrt RAM usage, to around 5GB (and I closed down firefox to make sure it didn't run out), only to go back down to 3.6GB at cycle four. I have no clue what happened there. Saves take about 8 seconds, but so far, I haven't had any crashes.

(it actually runs pretty nicely at 23FPS)

I'm gonna keep running it in the background, though it appears to leave enough memory free to actually do some work ^^

Mem usage at cycle 937 in spoiler -- the save started at 930.

Spoiler

After letting the game run for a total of 20 cycles, I've managed to break one of the steam turbines and had two dupes suffocate, but the game still runs nicely:

Spoiler

 

[Tonyroid]

On 9/8/2019 at 1:02 AM, Oxygenbreather said:

I'm around cycle 800 and my game keeps crashing. I love this game, but it just got unplayable that way. At some point it started to crash occasionally, then every few cycles, now I can't go through one cycle without it crashing at the morning save. The reason is IMO quite obviously RAM usage, whenever my system dings 8GB it or some sub process of it gets killed or fails to allocate or overflows stack whatever the detail is. Usually it crashes while trying to save.

I tried some emergency things, like killing lots of critters, removing debris as far as I can, removing pipes, clearing up regolith, nothing helps anymore, crashing periods get shorter and shorter.

So anything we can expect to be done about RAM usage of the game? Is there something I missed that would make a hugh difference?

I also already disabled anything from windows running in background that isn't absolutely necessary.

Since I searched and read a few topics when the original poster was about RAM, it's solely about RAM; I'm not talking about CPU or threads, or GPU. It's also likely not so much a memory leak, since it crashes already extremely soon. It's simply too much usage.

Why does the game use so much? IMO 8GB should be enough for a 2D game, no?

I have a BS in computer science and I have 10 years experience as a software development engineer in test. Using too much memory may slow the game down quite a bit but it's very unlikely to be causing the crash. The crash is most likely the result of a bug in the game itself. I've noticed that doing certain things are crash-y (wrangling shove voles, for example). It might be useful to think about what new things you started doing in the game when it started crashing. Things like stack overflows are related to memory but the CAUSE is a bug in the programming.

If it still seems like memory use is a key factor, then try testing the memory itself, I find that faulty hardware is more often the cause in situations like that.

Also, I tend to think that the game slows down due to processor speed more than lack of RAM. I don't see how the game could get by without processing every grid on every tick. I've written programs that do a similar "simulation on a grid" thing that use 16GB of RAM plus twice that in a swap file and the performance was still bound to the processor.

[xialeth]

18 minutes ago, Tonyroid said:

The crash is most likely the result of a bug in the game itself. I've noticed that doing certain things are crash-y (wrangling shove voles, for example).

OP notes that their save crashes upon morning saving, though, not during activities. Or is it something like wrangling shove voles during morning save?

I have noticed a slight increase in RAM used (~100MB) when it morning saves, but the game is running quite fine at 3.2-3.5GB mem usage (for the past 30 cycles, though I haven't been interacting much with it, and so far haven't gotten one of my ranchers to wrangle a shove vole wrangled a shove vole just fine)

[Chthonicone]

2 minutes ago, xialeth said:

OP notes that their save crashes upon morning saving, though, not during activities. Or is it something like wrangling shove voles during morning save?

I have noticed a slight increase in RAM used (~100MB) when it morning saves, but the game is running quite fine at 3.2-3.5GB mem usage (for the past 30 cycles, though I haven't been interacting much with it, and so far haven't gotten one of my ranchers to wrangle a shove vole wrangled a shove vole just fine)

Crashing during save data sounds like something locking the file preventing the game from saving. Have you made sure that your virus scanners are ignoring the save directory?

[Tonyroid]

9 minutes ago, xialeth said:

OP notes that their save crashes upon morning saving, though, not during activities. Or is it something like wrangling shove voles during morning save?

I have noticed a slight increase in RAM used (~100MB) when it morning saves, but the game is running quite fine at 3.2-3.5GB mem usage (for the past 30 cycles, though I haven't been interacting much with it, and so far haven't gotten one of my ranchers to wrangle a shove vole wrangled a shove vole just fine)

It could be any data anywhere that got corrupt for any reason at any time that ultimately causes a failure when the routine that saves the game is running, because that routine isn't expecting corrupt data or doesn't handle a particular situation. During saving is a time I would expect a crash after something has gone wrong, because making sure a save file isn't corrupt is REALLY important, otherwise your entire game could be lost. It would be a common practice to be especially careful with save files.

[xialeth]

2 minutes ago, Tonyroid said:

It could be any data anywhere that got corrupt for any reason at any time that ultimately causes a failure when the routine that saves the game is running, because that routine isn't expecting corrupt data or doesn't handle a particular situation. During saving is a time I would expect a crash after something has gone wrong, because making sure a save file isn't corrupt is REALLY important, otherwise your entire game could be lost. It would be a common practice to be especially careful with save files.

Otoh, the save file itself seems to be fine -- that's what I ran for 40 cycles without a single crash. So I have no clue what's going on, but I doubt it's RAM, unless something on OP's end is going totally haywire (as the save file is perfectly fine on my machine with 8GB of RAM, which the OP also seems to have)

[Tonyroid]

4 minutes ago, xialeth said:

Otoh, the save file itself seems to be fine -- that's what I ran for 40 cycles without a single crash. So I have no clue what's going on, but I doubt it's RAM, unless something on OP's end is going totally haywire (as the save file is perfectly fine on my machine with 8GB of RAM, which the OP also seems to have)

Yeah, that's what I'm saying. The save file should be great but the game may crash when it tries to save.

[xialeth]

6 minutes ago, Tonyroid said:

Yeah, that's what I'm saying. The save file should be great but the game may crash when it tries to save.

Ah, then I hadn't quite gotten your point the first time ^^' sorry for that. Any tips on what OP can do to save their save file?

[Tonyroid]

1 hour ago, Tonyroid said:

I have a BS in computer science and I have 10 years experience as a software development engineer in test. Using too much memory may slow the game down quite a bit but it's very unlikely to be causing the crash. The crash is most likely the result of a bug in the game itself. I've noticed that doing certain things are crash-y (wrangling shove voles, for example). It might be useful to think about what new things you started doing in the game when it started crashing. Things like stack overflows are related to memory but the CAUSE is a bug in the programming.

If it still seems like memory use is a key factor, then try testing the memory itself, I find that faulty hardware is more often the cause in situations like that.

Also, I tend to think that the game slows down due to processor speed more than lack of RAM. I don't see how the game could get by without processing every grid on every tick. I've written programs that do a similar "simulation on a grid" thing that use 16GB of RAM plus twice that in a swap file and the performance was still bound to the processor.

I tried some things and I don't see any smoking guns when it comes to the low frame rates and stuff that happens. My suspicions seem to be wrong.

[Chthonicone]

42 minutes ago, Tonyroid said:

I tried some things and I don't see any smoking guns when it comes to the low frame rates and stuff that happens. My suspicions seem to be wrong.

Do not forget that depending on memory conditions, page file sizes, swap space (for linux) they might be running out of allocatable memory, or at the very least, requesting it faster than the OS can supply it. This could also include swap/page file fragmentation causing issues where there isn't enough contiguous memory for a request. All of these games would force ONI to close, and appear to be a crash, when in particular, this was a planned termination because it could not continue.

This would also happen more frequently on low memory systems, and systems with low HDD space. It might be more frequent on systems that put the page/swap file on the HDD rather than an SSD as well, as the latter does not care about fragmentation.

[Oxygenbreather]

Well, first of all, I'm quite okay since I switched to Linux. Also the save file I provided is about 150-200 cycles past the moment where I threw my hands up Windows and since then, I worked on reducing it's complexity (like there are now much smaller cooling pipe circles than used to be, vacumed two areas, rebuild all stations and farm tiles (since they store all usages), and the regolith eating farm made good progress reducing those piles.

Yes, from Windows I only noticed that everytime it went from crash (and I had taskmanager running it was close the limit), so I just suspected Windows would have killed it, similar to what the Linux OOM-killer does in thight situations.

Also I disabled pagefile in Windows and Swap on Linux, since I consider my SSD too precious to be used for memory swapping.

As I had some crashes in Linux as well (always right after morning save (the save seems to be okay)).. Here I've seen it was not the OOM-killer, as if it would have been triggered, this would have been visible in dmesg.

Yes, it might be a "classical" coding issue, that just likely is triggered when memory gets scarce. Like using a memory block after it was free'd. I don't know. For one thing, I still just consider the amount of RAM usage quite .... surprising. As calculated before, it must more than several thousand 64 bit variables per cell... a few hundred I can understand, but several thousand...

[Stoned]

The crashes you run into happened to me on my last megabase (95 dupes, 2500 cycles). It started slowly, a crash every once in a while. Now it crashes in less than a cycle. I saw ram usage was high, so I upgraded from 16 to 32 gb, but no help. The crash log says last activity was 2 dupes getting a skill point, probably in the same game tick. cpu and ram usage goes through the roof and the game crashes. Makes sense that it only happens frequently when you have a ton of dupes.

Did you read out the crash logs? How many dupes are you running?

[Oxygenbreather]

I'm running 20 dupes. Didn't read the crash logs, most of the times later on, didn't even a crash catcher, but crashed straight to desktop.

The good news for me is, after some cleaning ups, my base no longer crashes at all (staying on Linux). However, after about 10 cycles I run into "terrible slowdown" which affects the whole system. Investigated, as usual, available memory according to "free" has become close to zero and kswapd is running the CPU like crazy (yes it swaps in and out mmap'ed data, albeit I don't have a swap file per se). Killed ONI system is responsive again. ONI had at the time of the great slow down: 6.1GB usage of reserved memory and 1.2GB of shared memory (according to top)

I stay with the assessment, the main issue is in my opinion, using a lot of RAM and in my humble guess, way more than would needed be. 

Crashes may come from other issues that are triggered in low RAM situation, whatever bug may be lay below. However, I believe even if fixed, it would only move the inevitable a bit further, as the kernel out-of-memory-killer would kick in soon after and likely target ONI at stay alive. Or the system just gets unresponsively slow as binary images get swapped in and out, and probably almost practically freeze altogether if there would be swap file (this is also why I don't have, I'd rather get some process killed than a practically freeze)