CPU Usage

[cezarica]

I used to run a vanilla world with Master and Caves for 12 players on a Linux server (Intel i5 2400 (6M Cache, 3.1 GHz) CPU and 8GB of RAM) and when switched to the beta branch I noticed that the CPU usage sky rocketed right away, even with just 1 player connected. It doesn't lag until get like 6-7 players and sometimes at 12 players for some is unplayable (CPU usage on one core is over 100% or that's what htop says).

Thanks to a friend moved temporary to his beefed up server (running Linux with E3-1246 v3 (8M Cache, 3.50 GHz) CPU and 32GB of RAM) and noticed that with only 1 player one of the cores that's running the Master shard is at 35%. When second player joined it jumped to like 65% and with 3 players it jumped to 72-73% ... What's going on? oO

When will server side see some much needed love and attention? i want to put some money aside for new hardware to run the DST server but with the results I'm getting right now I'm puzzled if my financial effort will be worth or not cos I'm not some prince in Saudi Arabia or something similar.

PS: Not sure if should have posted this into the [Don't Starve Together] Dedicated Server Discussion section (since it's about dedicated servers) or leave it here (since it's about the beta branch and the [Don't Starve Together] Dedicated Server Discussion is dedicated for the normal branch of the game), so posted here to get the attention of people that are hosting one in the beta branch of the game. Move if needed. Thanks.

[Zillvr]

This is just a guess, but I reckon the devs added in more debugging code to suss out issues in the beta branch (which will be later removed when full release) much like what they do in ONI beta branches. But that's just a hunch.

With regards to thread post location, I think you're right with putting it here. Since you did specify opting into the betabranch. Hopefully we can get more info from the devs soon on whether or not I had the right assumptions.

Could be entirely wrong though.

[CarlZalph]

This seems abnormally high from when I did my tests.

Do you have a modified server tick rate to be changed from the normal 15?

[cezarica]

No, tick rate is at 15, no mods, vanilla world (default settings).

I keep monitoring the server with Xenon CPU (mine is offline for now) and with currently 8 players the CPU usage seems to have stabilized a bit and is around 62% and from time to time jumps up to 72% shortly and returns to the 62-63% that seems to be the normal for now. Right now is Autumn and looking forward for a frog rain in spring.

Edit: Talked with my friend and he told me that he runs a server for another game (Garry's mod) but from what i saw in htop his app usage is a joke compared to DST servers. By joke I mean that uses like 4-5% at best with like 30 concurrent players.

The CPU usage since then has for some reason skyrocketed to 90-100% with 7-8 concurrent players.

Here's a htop output

and here's top output:

Still, with 6-8 players on my not too shabby i5 2400 it lags for some players and at like 10 for just about everyone. At 12 was unplayable for everyone and at 8-10 players a frog rain would make everyone moonwalk. CPU usage was similar, exception being that was around 90-100% after like 4-5th player, and host was going from green to yellow and then red as son as more people came in.

Bottom line is that i should start looking for better hardware cos clearly what i got isn't suited for running DST any more. I knew this moment will come sooner or later, was just hoping won't be this soon.

[Captain_Rage]

Isn't the default tick rate 30? As opposed to 15.

Have you tried fiddling with the threading options?

 

[CarlZalph]

On my i5-8300H 2.3GHz (Win10) with running a default setting caverns enabled server, both processes average at around 4% CPU when it's in idle mode.

When I join in the average usage goes to ~4.5% (1 player active, unpaused sim).  Jumps to ~7% when world is saving.

 

Assuming it's not a setting/mod related issue, then perhaps it might be a *nix-only issue?

 

1 hour ago, Captain_Rage said:

Isn't the default tick rate 30? As opposed to 15.

Nope, default tickrate on the server side is 15 for DST.

In fact when servers make it 30 it introduces some oddities for players who play in netbook mode by trashing the netvar stream.  I made a thread about it a bit back about it, not sure if it's still applicable or if it was fixed- changing the tick rate isn't something that seems too supported as tickrate 60 introduces weird locomotor jiggling.

Spoiler

 

 

[cezarica]

@Captain_Rage Haven't fiddled with anything, like i said everything is set to default, even the tick rate (it's set to 15). Will try with adding '-threaded_physics' but from what I read I have my doubts would help.

I copied the entire folder from my server to my friend's server and he said that it started to lag at 9-10 players, at 12 the host was red. One core that is running Main is constantly at 100%

Edit: As per Captain_Rage's suggestion i added that '-threaded_physics' and even if CPU usage didn't change at all, with 12 out of 12 people in host is yellow but overall the lag is somewhat a lot better. So thanks for the tip @Captain_Rage

Later edit: Like 30 minutes later and we are back to lagging.

[Captain_Rage]

It was just a wild shot. Admittedly I've only tried to set up a dedicated server once, ages ago, for testing purposes and it wasn't even running GNU/Linux. Hmm. What version of the Linux kernel are you using? Have you experienced anything similar in a different scenario on the same machine? Is the BIOS up to date? Sounds like something wonky is at work.

 

Edit:

Depending on your kernel version, an update or change of kernel might be worth a try. Not sure how to troubleshoot this, except for paying attention if the machine behaves in a similar manner in different situations.

[cezarica]

Well I suspected there was something wintin out camp causing high CPU usage but at some point all 8 online users all went to fight Klaus and even if host turned from yellow to green for a bit the CPU usage remained at 100% all the time.

Sometimes even with 4-5 people the CPU usage doesn't go down from like 90%. As is right now the only option would be to regenerate world and set the player cap to 8 or something, not that the CPU usage on a single core would change (based on current status).

Edit: Even now with only 3 people connected the CPU usage is sky high, so clearly something wrong is going on.

Second edit: Regenerated world on my server without that '-threaded_physics' option, and with no player connected I see a 12% CPU usage on Main (above) and another 12% for Caves on another CPU core. That's INSANE and makes no sense what so ever. Btw, [00:15:01]: Sim paused

Here's top output from my server with a fresh regenerated default world, started with that  '-threaded_physics' option (no difference with or without anyway), no players connected and world is paused. Help!

[Captain_Rage]

And what about the kernel version and/or BIOS? Have you tried updating or running a newer version of those? Are you running the 'normal' kernel of some distribution?

[cezarica]

BIOS version is latest (just rechecked on Gigabyte's website) and running Ubuntu Server 18.04.2 LTS with the kernel (right now uname -a says I have 4.15.0-34-generic #37-Ubuntu is running and if recall correctly there's a slightly newer version but should restart the system to kick in) that comes from them directly.

Later edit: I installed the Windows version on my Desktop PC (an i3-6100) and I can confirm CarlZalph's CPU usage, meaning that I get the same 4-5% for each shard on a fresh generated world with default world size and configuration, no mods and no players connected (sim paused). It seems it's something wrong with the Linux version since same thing takes double the CPU usage and from there onward it doubles until reaches 100% CPU usage.

Right now there are zero players connected (sim paused) yet CPU usage is like 54% on one core (above) and 12% on another (caves). This makes no sense.

root@Delta:~/coding# ps -eo pcpu,pid,user,args | sort -k1 -r -n | head -10
73.8 25295 steam    ./dontstarve_dedicated_server_nullrenderer -cluster MyDediServer -shard Master -threaded_physics
14.7 25298 steam    ./dontstarve_dedicated_server_nullrenderer -cluster MyDediServer -shard Caves -threaded_physics

I tried running a TheSim:Profile() and it crashed the server:

[03:16:42]: RemoteCommandInput: "TheSim:Profile()"
[03:16:42]: Profiler started.
[03:16:48]: [string "scripts/perfutil.lua"]:53: variable 'TheSime' is not declared
LUA ERROR stack traceback:
=[C]:-1 in (global) error (C) <-1--1>
scripts/strict.lua:23 in () ? (Lua) <21-26>
   t = table: 0xe0e9080
   n = TheSime
scripts/perfutil.lua:53 in (global) GetProfilerSave (Lua) <35-58>
   results = table: 0x1c581e30
   slotnum = 1
   session_id = 56DAA6E101C77F85
   onreadworldfile = function - scripts/perfutil.lua:40
   file = session/56DAA6E101C77F85/0000001353
scripts/perfutil.lua:107 in () ? (Lua) <104-111>
   results = table: 0x1c581e30
	[03:16:48]: [string "scripts/perfutil.lua"]:53: variable 'TheSime' is not declared
LUA ERROR stack traceback:
    =[C]:-1 in (global) error (C) <-1--1>
    scripts/strict.lua:23 in () ? (Lua) <21-26>
    scripts/perfutil.lua:53 in (global) GetProfilerSave (Lua) <35-58>
    scripts/perfutil.lua:107 in () ? (Lua) <104-111>
[03:16:48]: Warning: Widget:SetFocusFromChild is happening on a widget outside of the screen/widget hierachy. This will cause focus moves to fail. Is   ScriptErrorWidget       not a screen?
[03:16:48]: stack traceback:
        scripts/widgets/widget.lua:602 in (method) SetFocusFromChild (Lua) <599-624>
        scripts/widgets/widget.lua:621 in (method) SetFocusFromChild (Lua) <599-624>
        scripts/widgets/widget.lua:621 in (method) SetFocusFromChild (Lua) <599-624>
        scripts/widgets/widget.lua:649 in (method) SetFocus (Lua) <626-658>
        scripts/widgets/menu.lua:83 in (method) SetFocus (Lua) <74-85>
        scripts/widgets/scripterrorwidget.lua:109 in (method) OnUpdate (Lua) <102-119>
        scripts/update.lua:90 in () ? (Lua) <33-129>

[cezarica]

Just copied the files as they are from my Linux server to my Desktop PC and started the servers. In the image below is the CPU usage:

No players connected on Linux and it's CPU is at like 70-80% for main and at like 13-14% for caves. What can justify the ridiculous CPU usage while running the same world from Linux?

Should I consider installing a virtual machine on Linux just to install a copy of Windows and host the world there?