Periodic system freeze on linux

There is a related thread here. https://github.com/ValveSoftware/GameNetworkingSockets/issues/165

I was able to greatly increase the time it took to freeze by closing all other applications. But it still happened eventually. Usually the other applications I have running are some chat programs and a browser. I think all of this points to thread starvation. The browser uses a lot of threads so closing that would likely result in delaying the issue. The ram and cpu usage do not increase during this issue either so it can not be either of those.

This sounds like a swapstorm. When it first starts to lag out go check out memory usage, I suspect you'll find your system is getting close to OOM and spending way too long trying to reclaim pages. I like to use cgroups, my friend recommends earlyoom as an easier out-of-the-box solution.

I tried that, memory use did not increase at all during the issue. It is possible the terminal freezes at the same moment the issue starts and thus does not show the latest data, but it seems unlikely. I would expect the memory use to spike before symptoms start and only to see a slowdown when it started to get too high. Also the machine remains partially responsive for up to a minute once the first lag spikes hit so one would expect the terminal to get a few updates in.

Having said that I could be wrong, and a swap storm was my first though too, so I will check it again.

 

edit: Ok that was kind of inconclusive, the ram usage never fluctuated by more than 1% the whole time I was playing from loading the save until the crash. But the kswap thread was running at 26% of a core so there was some swapping happening. Also worth noting, the cpu usage of ONI dropped to almost nothing.

I will adjust the swap settings and see what happens.

edit2: So I learned how to correctly interpret top output today, the mem% stat only includes actual ram used and does not include swap memory. The issue is indeed a swap storm. ONI reached 14gb total memory usage by the time it crashed. Is this normal?  Does everyone who plays this game have way more ram than me? I would be the first to admit that 8gb is a bit lean for a gaming rig but this seems a bit extreme.

Edited December 13, 2021 by DuncanR

If kswapd is using ANYTHING appreciable then you've got RAM problems. 26% of one core's time is being spent just shuffling pages around trying to free up enough RAM for a malloc() to succeed or a page-in to happen! That's a lot of wasted cycles, and that's just the time spent trying to get the pages, it doesn't count all the processes that were trying to do things that needed RAM that are instead just sitting there waiting for pages to be available (processes like oxygen not included and your window manager).

 

Looks like I have same issues on Ubuntu 20.04 kernel 5.11.0-41-generic

May be related - exiting to main menu decreases memory consumption of game process by about 500 Mb, the other 5+ Gb are freed only after exit to OS, and initial memory usage of main menu is about 1.5 Gb

Some update: My build as reported in main menu is u39-490154-5

On mem usage via glances utilty

Memory usage main menu after brand new start: 1.50G, 3.70G of free memory

After exit to menu (30 min of gameplay): 5.32G, 440M of free memory

After exit to desktop: 0, 5.74G of free memory

As far as I can recall it was about 250M memory left when I exit to main menu, I didn`t dare to take screenshots

 

55 minutes ago, kirill.stepanov said:

As far as I can recall it was about 250M memory left when I exit to main menu, I didn`t dare to take screenshots

that is not only at linux, there is indeed extra memory usage when you go back to main menu

i not bothered talk about that before when i debugged, but there is running allot gameobjects after you go back to main menu

Edited December 14, 2021 by gabberworld