Yes, I have a similar issue! With me only happens in someone's world, if I host the game or if Klei hosts it runs normally.
I've reported my bug with picture in the link below.
If I try to play someones world and enter in a cave, the games crashes during the loading page. It says the server is not responding and I am disconnected from it. I am able to rejoin the same server, but doesn't matter how many times I try to enter in a cave, it will crash.
If I am the host or if I play in a Klei official server, I am able to explore caves normally.
I've asked in different servers I play and all the players can load caves, except me. Usually I pick the best connection servers, so I don't believe is an internet issue. Even Klei official servers sometimes presents poorer connection than some players, but I still can play caves if Klei hosts.