Profiling your server - redux - I need You!

[bizziboi]

In the past week the profiler has been modified a fair bit to be able to better analyze the behaviour of dedicated servers, and since some servers are behaving quite erratic I could use your help more than ever.

 

If you have a dedicated server, I would love to see your profiles under load. I am looking for Windows and Linux profiles, of servers that behave well as servers that behave bad. It may help to spot common patterns that cause issues.

 

If you're interested in getting performance stable, like I am, you could help out by grabbing a profile and uploading it here - please also mention some details hardware, OS and whether you are running from a dedicated hosted server, or a spare PC of your own.

 

If the game was behaving particularly erratic, additional details will help (running mods? Winter?). I log the number of players and pings so you don't have to give me those.

 

To run the profiler, open the console, and execute the command 'TheSim:Profile()' as administrator as a remote command (tap ctrl when you type the command in the in-game console).

 

This should create a file 'profile.json' in your server's document directory containing 3 seconds worth of profile data (it's kinda large, probably close to the 2mb mark, and, as @Kzisor pointed out, please zip it up as the forum won't accept json attachments).

 

Your server will lag during these 3 seconds (not freeze).

 

Help me make this game better!

 

Kaj

[Kuldiin]

Name:-                                 (Kuldiin) United Kingdom

Host:-                                   http://jestservers.com/about/

 

Day:-                                     524

Mode:-                                  Endless

Season:-                               Winter

Players:-                               2

Mods:-

 

--Haruz

--Mitsuru

--Wolf

--RoG Characters

--Auto Retrap

--Simple Protection

--Picky

--Wormhole Marks

--Bee Nice

--Renew World

--Burnie

--Strong Traps

 

Okhost2ppl190215.zip

[Kuldiin]

Name:-                                 (Kuldiin) United Kingdom

Host:-                                   http://jestservers.com/about/

 

Day:-                                     532

Mode:-                                  Endless

Season:-                               Summer

Players:-                               5

Mods:-

 

--Haruz

--Mitsuru

--Wolf

--RoG Characters

--Auto Retrap

--Simple Protection

--Picky

--Wormhole Marks

--Bee Nice

--Renew World

--Burnie

--Strong Traps

Badhost190215.zip

[ToNiO55]

Hi @bizziboi,

 

Day: 1556

Gametype: Wilderness

Players: 8 (one player join with high ping - look screenshot)

Season: Summer

 

Mods:

 

- Renewable World

- Nature's Still Wild

- Respawnable Gears

- Simple Protection

- Global Player Icons

 

My Server Dedicated:

 

Debian 7.8,Intel Xeon W3520, 24Gb, Kernel 3.2.0-4-amd64, 100Mbps,2 To HDD

 

http://kfkserver.com/download/dont_starve/profile.json_3.rar

 

 

[Kuldiin]

Heres my stats also.

 

The CPU is never over 20%, yet the server is struggling?

[bizziboi]

Hey @Kuldiin, 

 

The server info panel is really not a good measure of what is being used. The game runs (mostly) singlethreaded, and the panel shows the sum of multiple cores. 

 

Also, things like hyperthreading can affect performanc Ingame profiles are really much more useful - rest assured I am working on optimizing, and also looking into multithreading certain parts - unfortunately, the lua moddibility of the game makes it hard to thread a large part of where the execution time is going, so I have to find it mostly in other spots.

[Kuldiin]

Thank you for the reply

[Kuldiin]

My server was starting to get very bad, then I had an idea. What if I got rid of all the spider dens throughout the world?

 

After clearing about 70 dens, and 40 spiderqueens (and all the spiders), and all Beefalo, my server went from 'Bad Host' at 1-2 players, to 'Good Host' permanently with 4 online.

 

Not had a chance yet to test the server open to the public, but it seems a good spider culling helped my server immensly.

 

To do the above, I went godmode, then teleported between each nest torching as I went.

[Kzisor]

@bizziboi, here is my newest profile, it's taken time to get enough players on the server to really stress it.

 

- Day 8

- 12 out of 12 players

- CPU: i7 2600k @ 4.4GHz

- Core Usage: 50%

 

[bizziboi]

@Kzisor,

 

Thanks. Yep, your (beefy) machine runs that game (11 players actually according to my profile, I guess the host counts as 12?)  at a very solid 60fps. It has about 55% frametime on average to spare.

 

@ToNiO55,@Kuldiin,

 

So this is where it gets confusing:

 

Kzisor's machine runs the game at about 130fps with 12 players. Your servers run the game at sub 30 with 6 players. Looking at the profiles the exact same code runs about 10 times as slow - and I'm fairly sure his machine is not running 10 times as fast.

 

I have 2 dedicated servers, one runs the game at about 120fps with 6 players, the other barely makes 60 with 4 players- the one at 60 has better specs than the one at 120fps. Both run the game substantially worse than my desktop I5.

I am, as mentioned, looking at threading parts of the code, but I have no control over what else happens on the server - it is after all a shared machine, and I suspect the way the hardware is shared is what makes certain servers run like a dream and others run like...not, a pleasant dream. More entities will expose this more readily, but at the same time, the servers that run well still run well with a lot of entities, so that's only part of the story - the system as a whole runs slower.

 

I am investigating - for sure.

[Kzisor]

@bizziboi, let me make things a little more clear. I actually have 3 instances of the dedicated server running on my computer; it is my personal computer.

 

For good measure here is Server 1 profile:

 

-- CPU: i7 @ 4.4 GHz

 

-- Server 1

-- Day 13

-- 6 out of 12 players.

-- Core Usage: 32.7%

 

-- Server 2

-- Day 2 (paused)

-- 0 out of 12 players.

-- Core Usage: 2%

 

-- Server 3

-- Day 83 (paused)

-- 0 out of 6 players

-- Core Usage: 2%

 

[RFiD]

Hey here is the profile i promised to upload in that thread: http://forums.kleientertainment.com/topic/51128-ping-limit-ping-kick-for-dedicated-servers/

 

Server is running on killservers.com

 

5/6 Players

pings were decent

 

dont remember the day.

 

 

 

profile.zip

[bizziboi]

Hey @RFiD, 

 

Thanks, and yes - the server is running at about 40FPS (so it should show as OK host there) with 4 players. I'll have to investigate it more, that was just at quick glance. May I ask where your server is located?

 

Thanks,

Kaj

[Kuldiin]

Would a copy of my world now I eliminated Beefalo/spiders be of any use?

Ill try today with a public server see how many people I get on.

[bizziboi]

@Kuldiin,

 

Hmmm, given the amount of mods you have running it's unlikely the world would run on my system. I can see what's in your world though - I store a copy of it in the profile so I can examine the amount of entities etcetera so if you grab another profile I'll be good to go.

[ToNiO55]

@ToNiO55,@Kuldiin,

 

So this is where it gets confusing:

 

Kzisor's machine runs the game at about 130fps with 12 players. Your servers run the game at sub 30 with 6 players. Looking at the profiles the exact same code runs about 10 times as slow - and I'm fairly sure his machine is not running 10 times as fast.

 

I have 2 dedicated servers, one runs the game at about 120fps with 6 players, the other barely makes 60 with 4 players- the one at 60 has better specs than the one at 120fps. Both run the game substantially worse than my desktop I5.

I am, as mentioned, looking at threading parts of the code, but I have no control over what else happens on the server - it is after all a shared machine, and I suspect the way the hardware is shared is what makes certain servers run like a dream and others run like...not, a pleasant dream. More entities will expose this more readily, but at the same time, the servers that run well still run well with a lot of entities, so that's only part of the story - the system as a whole runs slower.

 

I am investigating - for sure.

 

I think this is linked to my mods which spawn all and anything on the map,

the lags are becoming more intense with much days, this is probably what causes the fall of fps, but im not sure

so when I have start the server for the first time, there was no big lag as now even with high ping of player, whether the day or night, summer or winter

 

if you ever need another profile does not hesitate

 

I forget to specify the processor of my dedicated server:

Intel Xeon W3520

 

[Kuldiin]

Heres my latest. Running as Good Host, but with the changes above.

profile.zip

[bizziboi]

Hey @ToNiO55, 

 

I will definitely look at the entity counts, and if you start another world - I would love a profile of it when it starts out but has some players, and when it gets older and gets bad with a few players.

 

@Kuldiin,

 

Thanks, appreciate it. I will look at that one as well  

[RFiD]

Hey @bizziboi

 

Most of the time the server were running fine. even with 5-6 players, It just happen sometimes and lasts till a certain player leave.

 

 

The server should be located in Nottingham,UK - as stated from killservers.com

When searching for the IP via google it say its located in the US.

 

Server IP: 167.114.99.91

 

People from the UK have pings between 15-35 while I play from germany with ping 45

Players from the US have pings +150

so the serverlocation in UK (as stated from killservers) should be correct. 

 

If you need any more info or data to work with, I will gladly provide. 

 

 

Edit:

Ok today i searched for the IP location again -  and now its located in Canada ? °_O

Crazy changes

[ToNiO55]

Hi @bizziboi,

 

sorry for the delay

 

as you requested,i have start with a new map and with same mods

 

summer, wilderness, twilight, 39 days, 8 players

 

if you need more information, do not hesitate

 

http://kfkserver.com/download/dont_starve/profile.json_4.rar

 

[RFiD]

Hey 

took another one today.

 

6/6 Players - good pings - strong lags for everyone

 

I think around day 27

 

the lags came in different intervals and lasted about roughly 30sec - 90sec

they showed up first early after server was restarted.

 

Edit: Some Players think it could have something to do with spiders. they said everytime they went near alot of spiders it started lagging 

 

 

profile.zip

[afetogbo]

Well I am not permitted to upload that kind of document. profile.7z

 

I have these mods running on my server most have been edited and are private. Downloading from the workshop will not work. http://steamcommunity.com/sharedfiles/filedetails/?id=386193489

 

I only use map revealer for testing and it was disabled for the profile I did.

 

I am not sure what I have to do to post files but I can use google drive or email instead if needed.

 

Oh I also made this topic to document my struggles and achievements.

http://forums.kleientertainment.com/topic/51269-adventures-in-afetopia-a-heavily-modded-and-customized-dedicated-server/?p=615378

[jrose78]

Running from my main rig. I play the game from this machine as well. I am running the dedicated server software.

 

Profile took with 16/16 player on Wilderness

 

Specs:

Intel i5-4670k

16GB Ram

Windows 7

profile.zip

[Kzisor]

@bizziboi, here is my newest profile.

 

Running Servers:

-- Ysovuka - Paused

-- Ysovuka2 - 17 out of 24 players

-- Ysovuka3 - Paused

-- Lan - Paused 

 

Ysovuka2 Stats:

17 our of 14 players

 

-- CPU Information:

    Current Threads: 12

    Usage: 11%

    Average Usage: 11.48%

    Cycle: 12

    Average Cycle: 11.59

 

-- Memory:

    Commit (KB): 444, 448

    Working Set (KB): 445, 284

    Sharable (KB): 9, 948

    Private (KB): 435, 336

 

-- Network Traffic:

    Send (B/sec): 116, 000

    Receive (B/sec): 8, 500

 

Profile:

[afetogbo]

Dedicated server: Afetopia V1.2

WiFi connection to router (my cable broke)

Client on same computer

Players 1/16

80+ mods

i5 8gbram

 

https://drive.google.com/file/d/0B1X67ezxiwCCaTdXU2txZ0NCN3M/view?usp=sharing