On multiple servers I've been unable to enter caves in someone else's world. The loading screen will start and then game will kick me back to title screen. Then will crash completely and close while trying to browse games. I was playing in split screen mode on every instance and it kicked us both.
I mentioned this in a game I was hosting and others had mentioned they had the same issue.
Try to enter caves in another hosts world while in split screen mode.