Other player disconnects completely after going into caves

After going into the caves, they completely lose connection to the server and they get "Whoopsie! Bad password" pretty much every time to rejoin. How do we deal with this? We used rollback and it worked, but it kept happening again. As a host I didn't have issues with the caves at all. 

Thanks!