Character movement doesn't work when joining other worlds

I like to play DST with my friend on a weekly basis, but we've found that whenever I join a server hosted by him or anyone else, my character glitches out, walking in different directions automatically and teleporting backwards--obviously quite inconvenient when hunting or fighting monsters.  The teleporting is also visible on his screen, so I know it's not an issue with my display.  The first time it happened we were using mods and so thought it might be related to that, but even when he created a new world with all mods disabled (and I have no client mods enabled), the issue persisted.  Same thing happened when I joined a public server.  However, when I'm playing in a server I host myself, there's no problem.