So I'm not sure if this is a bug or not but I figure I'd post it just to be safe
When you go into a cave with followers your followers attempt to return home in the case of pigs, merms, and spiders or wander away in the case of unattached spiders which means the change to keep followers if they're close enough when you disconnect or go into the caves doesn't work properly because they'll move away from the spot in seconds.
1. Recruit a follower
2. enter caves or disconnect on a dedicated server