Jump to content

Rethinking Action Prediction


Recommended Posts

Context

Rubberbanding is a huuuuge problem. It's particularly frustrating because there are several scenarios where the player can tell that they aren't where they think they are on the server, but the client hasn't figured it out yet. I know you're probably working on ways to improve that situation, but I want to bring a little perspective from other networked games I've tried.

 

Path of Exile is an amazing game. Not for everyone, but it has incredible depth to its strategy and interesting art, and many other great things. I love it a lot. But there's one thing in it that always pushes me away: desync. See, Path of Exile uses the same networked system where the client predicts what your character will do with your input, and shows that before it hears back from the server. This results in a state of constant desync -- usually you're only a few steps behind where you appear to be, but sometimes you're caught in the previous room because you ran into the doorframe instead of through the doorway, or a monster got in the way, preventing you from escaping that group. Desync is the single greatest problem with that game.

 

I'd really really really hate to see this happen again. My heart is breaking right now over this. While action prediction has some upsides, such as making the game feel more responsive, I'd argue that the costs are much greater. Once the player becomes familiar with the way it works, it instills a constant fear that the client is misrepresenting the server's reality in a fatal way. I'm running away from that spider, but am I actually away from it? Nope, I just rubberbanded back into it and died. This is a particularly frustrating situation because for a time you thought you were succeeding at something, but then suddenly you find out that you weren't doing anything on the server at all.

 

Alternatives

Now, there are two alternatives to action prediction:

1) Trust the client. If it doesn't look like the player will get hit to the player, then they won't get hit.

2) Wait for the server. Don't do anything on the client until the server says so.

 

I argue that in the case of Don't Starve Together, both of these alternatives are better than action prediction.

 

Trust the client

The primary disadvantage of trusting the client is that it opens up opportunities for cheating. This is a pretty big disadvantage for some games, but in Don't Starve there was always the option to cheat. The host can continue to cheat all they want. So clients cheating is much much less of a problem in DST than it is in a MMO/ARPG.

 

What this would look like in DST is that if you're too far away from that spider on your client, then the spider can't hit you. To prevent things like the server saying "spider, do an attack" and just having it miss outright because you're too far away, you can alter the communication a bit to have it say "spider, aggro player A", then the client handles the rest. You might see something different on a different client, but what really matters is that the player who's being most affected is able to react to the situation despite lag.

 

Wait for the server

The main disadvantage of waiting for the server is that the game feels less responsive. However, in low-lag scenarios it's just fine, because the lag in responsiveness is very very small. In high-lag situations, you have a large gap between clicking and acting, but at least you know it will do it at some point in the future and you're not being tricked into thinking the lag is less than it is. With action prediction, you'd think you'd be moving, and then you'd snap back -- rubberbanding.

 

What this would look like is if I say move to the right, I don't see it on the client until I move to the right on the server. This is much more manageable for a player because in the worst case, the client is just an outdated version of the server's scenario, instead of trying to predict into the future and then being wrong. Yes, it makes the game feel less responsive, but the upside is that you aren't second-guessing the client constantly.

 

Conclusion

I know that action prediction is already baked into the netcode of the game at this point. But I hope it's not too late to reconsider. Action prediction has some nice things, but it also has its own very serious downsides, which in the case of a game in which death is supposed to be very punishing, are magnified.

Link to comment
Share on other sites

I'll wait for the latency updates in the next few patches Klei is working on before further comment from me here. 

 

I want to be in your boat on the hopeful note, but in Path of Exile's beta and alpha everyone was also hopeful, "Desync will be fixed!" "Desync, just beta probs". Now it's a couple years from its release and desync is still a huge problem. They added more resync conditions and that helped a bit (mostly "resync-on-stun"), but I've still ripped multiple hardcore characters to desync, and had many many softcore deaths and much frustration due to it.

 

I want to believe, but I'm afraid of being wrong.

Link to comment
Share on other sites

I want to be in your boat on the hopeful note, but in Path of Exile's beta and alpha everyone was also hopeful, "Desync will be fixed!" "Desync, just beta probs". Now it's a couple years from its release and desync is still a huge problem. They added more resync conditions and that helped a bit, but I've still ripped multiple hardcore characters to desync, and had many many softcore deaths and much frustration due to it.

 

I want to believe, but I'm afraid of being wrong.

Just waiting to see what to comment on specifically. Realistic on the potential results. 

Link to comment
Share on other sites

  • Developer

Hey guys, there's still a lot of improvements we are actively working on to minimize the rubber banding, but I did want to let you know that prediction can actually be enabled or disabled dynamically by the client. In fact, ghost mode automatically disables it right now. We will definitely expose this as an option for the players to decide if they want to turn it on or off if that gives them a better experience.

Link to comment
Share on other sites

I want to be in your boat on the hopeful note, but in Path of Exile's beta and alpha everyone was also hopeful, "Desync will be fixed!" "Desync, just beta probs". Now it's a couple years from its release and desync is still a huge problem. They added more resync conditions and that helped a bit (mostly "resync-on-stun"), but I've still ripped multiple hardcore characters to desync, and had many many softcore deaths and much frustration due to it.

I want to believe, but I'm afraid of being wrong.

Sadly, in PoE there will always be desync. It's like a gameplay mechanic at this point. Learn to see the signs (mobs attacking something invisible elsewhere, suddenly taking damage etc) and react by logging out. Or don't play with skills like cyclone etc. The community can keep beating the dead horse or deal with it/play another game...

Yesterday I played DST on a server with only two cases of rubberbanding in an hour or so, hopefully today brings another almost rubberbandfree day :). It already makes DST so much more enjoyable than a few days ago..

Link to comment
Share on other sites

I have to say IMO you're right, there's no reason not to put more data to client side here, or send more data, being that 1. server is not klei server but player-side anyway 2. players can hack code or use console anyway 3. even if cheating wasn't that easy its almost irrelevant in a DS universe.

Link to comment
Share on other sites

Anything to make game run better. So far the lags and rubberbanding are killing all the fun. I don't want to play, and I don't play that often for that reason. Game is so laggy I can't kite single spider. Tried everything. It is laggy even if you are close to the server and have low ping.

Link to comment
Share on other sites

I just got a key last night and was able to play for 3 hours. The rubberbanding gets annoying, I left the first server because it was happening every 15 seconds or so. The next server was better, but it would happen every few minutes.

 

After playing I checked Twitch for some streams, and it turns out *every single* non-passworded server at the time had someone streaming *and* hosting the game, except for like 1 ( so only ~6 open at the time as it was late). These people are obviously using up lots of computer resources as well as network resources on streaming as well as hosting DST. Not to mention your average person has between 1 and 4 other people in their house and who knows how many of them are watching Netflix, torrenting, Youtubing, etc.. Rubberbanding every so often makes some sense considering any joe schmoe is hosting, and every one that is hosting is also streaming.

 

I think the biggest "fix" we'll see is dedicated servers. The action prediction mentioned earlier seems to work well for movement -- the game movement runs OK for me when it's not rubberbanding because I can tell a big difference in between being alive and ghost mode which is constantly "laggy" and slow moving. Other than the normal delays in combat due to ping I wouldn't be surprised if every rubberbanding event I had was due to one of the hosts roommates pulling up a new Youtube video while another was uploading 10 new selfies to Facebook. Combat is another story though since monsters are attacking where you are on the server and not where you see yourself on the screen, and with these sketchy hosts with spiking pings it can be a pain to time things right even considering this (such as starting to run from a spider before you see them start to attack since you know on the server they are already starting to attack).

 

What are the bandwidth requirements for hosting a game of 4 players?

Link to comment
Share on other sites

I just got a key last night and was able to play for 3 hours. The rubberbanding gets annoying, I left the first server because it was happening every 15 seconds or so. The next server was better, but it would happen every few minutes.

 

After playing I checked Twitch for some streams, and it turns out *every single* non-passworded server at the time had someone streaming *and* hosting the game, except for like 1 ( so only ~6 open at the time as it was late). These people are obviously using up lots of computer resources as well as network resources on streaming as well as hosting DST. Not to mention your average person has between 1 and 4 other people in their house and who knows how many of them are watching Netflix, torrenting, Youtubing, etc.. Rubberbanding every so often makes some sense considering any joe schmoe is hosting, and every one that is hosting is also streaming.

 

I think the biggest "fix" we'll see is dedicated servers. The action prediction mentioned earlier seems to work well for movement -- the game movement runs OK for me when it's not rubberbanding because I can tell a big difference in between being alive and ghost mode which is constantly "laggy" and slow moving. Other than the normal delays in combat due to ping I wouldn't be surprised if every rubberbanding event I had was due to one of the hosts roommates pulling up a new Youtube video while another was uploading 10 new selfies to Facebook. Combat is another story though since monsters are attacking where you are on the server and not where you see yourself on the screen, and with these sketchy hosts with spiking pings it can be a pain to time things right even considering this (such as starting to run from a spider before you see them start to attack since you know on the server they are already starting to attack).

 

What are the bandwidth requirements for hosting a game of 4 players?

 

I've been playing for a week now, and ping has not been a good indicator of whether rubberbanding occurs. The most reliable thing has been the distance between me and the host player, oddly.

 

As for bandwidth requirements, it looks like you receive 1-4kbps as a client, so multiply that for the server by the number of players. 16kbps is well within bandwidth requirements, that's about as much as some really low-quality audio right there (128 or 256 kbps is high-quality audio). Granted, interruptions in that will be a bigger problem than interruptions in properly-buffered audio being streamed, but it's still a pretty small amount.

 

But yeah, maybe dedicated servers will be an improvement. The problem is that the most common use case for Don't Starve Together will be a few friends going "Hey, let's play a game together" -- one of them hosting, and the rest joining. So if this rubberbanding situation is going to be the case there, then that's a major problem.

 

Edit: Managed to get a 4-person game going that I was hosting. It looked like it peaked around 20kbps sent.

Link to comment
Share on other sites

Not sure if it'll be of any use, but I wanted to comment on rubber-banding in the game. I've played DST with someone locally, in the same physical room, and there still was lag for them (I was host).

 

I'm not 100% sure on how the client-server code works within DST, but here's how I dealt with synchronization and such in my DS mod:

 

Unlike DST, we had a dedicated server for all games... although at the time we only had one server running at any given time, the plan was to modify the code so one server could host multiple games.

 

The networking of the mod was quite simple. The server acted as a relay for the clients. If two people connected to the server, then each instance of DS would be its own master for location and action information for that player. The server would simply relay any communication required to other connected clients.

 

So for example, If I were to click on the ground somewhere to move, the game would send that click location to the server, and the server would relay that location to the client, in effect moving me as needed.

 

Every now and then, there used to be a jump in player movement. Say for example, both players were walking somewhere, each player would see the other warp ahead every minute or so. I came to find out that this was due to the differences in land speed of players. There are mechanics in the game which allow players to move faster/slower based on various conditions, one being a road walking speed increase.

 

In our first synchronization of movement, we overlooked those land-speed differences which in effect led to the desynchronization of players, because the clients weren't moving the other players to the point received fast enough. Once these were accounted for, most of the lag disappeared, and what was left only remained due to synchronization of the entire map.

 

Regarding preventing cheating, there 'are' ways to make cheating more difficult in Don't Starve (to the point of making it impossible), and in turn DST. I've thought about such methods quite a lot while developing my mod, but never got around to implementation.

 

Anyways, enough of me.

 

My 2 cents.

 

-Crynux

 

 

Link to comment
Share on other sites

@Crynux That sounds essentially like a "trust-the-client" model. Which would be great, I think (it makes lag only relative to other players, not the rest of the game world), but it's not what we have, unfortunately.

 

Edit: That being said, I've been experimenting with action prediction turned off. It's definitely a different experience, but I think one that's less frustrating in general in bad conditions. Hopefully the lagginess will go down over time and make the difference less salient.

Link to comment
Share on other sites

@Crynux That sounds essentially like a "trust-the-client" model. Which would be great, I think (it makes lag only relative to other players, not the rest of the game world), but it's not what we have, unfortunately.

Edit: That being said, I've been experimenting with action prediction turned off. It's definitely a different experience, but I think one that's less frustrating in general in bad conditions. Hopefully the lagginess will go down over time and make the difference less salient.

I just look to other games as examples.

If FPS games, fighting games, and even other indie titles can get latency to acceptable levels, then DST can too.

Link to comment
Share on other sites

I just look to other games as examples.

If FPS games, fighting games, and even other indie titles can get latency to acceptable levels, then DST can too.

 

Go join a server and turn off action prediction:

ThePlayer:EnableMovementPrediction(false)

Then move around.

 

The latency isn't in the network. I tested it on a server with consistent 100 ping and got what felt like 500+ms of delay. Most titles in the genres you list use extremely efficient compiled languages as their primary world-simulation (e.g. C++). All of the world simulation in Don't Starve is done with interpreted Lua. This means that it's essentially "compiling on the go" as it's running the code. It may be possible to improve the response time of the server, but with this situation it looks really hard to me.

 

Edit: sorry, wrong function name. MovementPrediction, not ActionPrediction.

Link to comment
Share on other sites

I wouldn't necessarily blame Lua for the lag... I mean games like Minecraft have decent multiplayer capabilities and use Java, which in my opinion is one of the worst languages ever (even though it is somewhat 'compiled' into Java bytecode... it runs like a slug compared to anything else I've seen/used).

 

If I remember correctly, the devs have built the game so that a portion of it is written in C++ (or at least something compiled). From what I can tell in looking at the Lua code (for about 5 minutes)... it doesn't seem that the networking is being done in Lua, but in the compiled binary that is the game engine (i guess you would call it that... my game lingo isn't up to date). Even so, I don't believe the interpretation of Lua is much of a drawback.

 

When I was creating my mod (I know, I refer to it a lot these days), at first we were using HTTP POST/GET requests and responses for networking. This worked decently fast, but we changed it once I found out how to hack sockets into the Linux version of DS. I'm sure using HTTP POST/GET from Lua interpreted in a binary in a mod is much slower than a networked binary interpreting Lua.

 

At this point, I believe that the lag is primarily a issue of optimization. There's quite a lot of data to synchronize in the game... I'm not entirely sure how the devs do it currently, but I'm sure that there are things that can be changed to decrease network traffic if needed (there almost always is).

 

We have to keep in mind that the game was designed to be a single player game. Adding multiplayer, whether you're the devs or not, isn't necessarily an easy task. There are multiple decisions to be made with regards to how to synchronize players, data, mobs, events, all of which have their own requirements, and each decision has it's own performance impacts and considerations. Some of these decisions will be changed or refined as time goes on, and in theory, our suggestions can aid them in changing for the better!

 

Nonetheless, I still opt for a stand-alone dedicated server... it'd take the extra load off of the current host, and put it elsewhere. Which may in itself be the solution to lag.

 

Another 2 cents.

 

-Crynux

Link to comment
Share on other sites

There's quite a lot of data to synchronize in the game... I'm not entirely sure how the devs do it currently, but I'm sure that there are things that can be changed to decrease network traffic if needed (there almost always is).

 

Nonetheless, I still opt for a stand-alone dedicated server... it'd take the extra load off of the current host, and put it elsewhere. Which may in itself be the solution to lag.

 

I feel like this will help a ton. It might depend on the computer, but I notice lots of rubberbanding events related to the host doing regular in game things such as reviving themselves, reviving others, using wormholes, fighting hounds, a new player joining, etc.. on top of rubberbanding from alt-tabbing, and I assume any time the "saving" dialogue pops up as the game freezes for a second. Having said that, there is a server I've been on ~3 times for ~6 hours total now that is consistently good, consistently low latency (note, not just the listed ping but "real world" latency of picking bushes/combat etc.., as someone else mentioned even with decent ping some servers feel like 500ms delay), and very few rubberbanding events at once every 5-10 minutes rather than once a minute or more. I don't mind regular latency so much, but rubberbanding makes any combat unplayable, or getting to a fire to stop freezing/darkness.

 

I don't know if the amount of data being sent/received is much of a problem. With 4 players it seems like the host doesn't need to upload more than 50k-70k a second peak. The only server I've had rubberbanding due to packet loss was someone streaming twitch and I assume they set their bitrate to the maximum their connection can handle to get more viewers from a clear stream which doesn't leave enough left over to host a game. I've kept an eye on the packet loss in other servers and it's pegged at 0.0% during all rubberbands.

Link to comment
Share on other sites

  • Developer

Lots of good ideas here. Decreasing network traffic is most certainly one of the steps we will be taking to reduce the effects of rubberbanding as we continue to improve on the prediction itself. For LAN games, it may make sense to default to prediction off, as we have done in our early LAN demos. As for dedicated servers, we're working on it ;)

Link to comment
Share on other sites

From my experience better servers/host will massively reduce rubberbanding but, I don't believe it will have enough effect to make the combat workable. I've been play with friends with pings consistently below 40 and my friend hosting on his haswell i7 gaming rig and combat is still nearly impossible. For example with Koalephants in single player I can get in 5 hits between attacks without getting hit. In DST right now I will often get hit after only attacking once. Another example is spiders. In singleplayer stun locking an individual spider without getting hit is a breeze but in DST I almost always get hit before the server registers I'm hitting it. In a game where combat is so focused on timing even if you could get the lag down to a negligible difference from ping 50 to 200 ms is too much lag. IMO anything but trust the client architecture will feel unacceptably laggy and make the game far less enjoyable. In a game that is intended to be player vs environmen I don't understand why you wouldn't want as much on the client side as possible. Yes people could cheat but that is hardly a big deal unless the goal is to create a primarily PvP experience.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

Please be aware that the content of this thread may be outdated and no longer applicable.

×
  • Create New...