Ballphins from player-built Ballphin Palaces sometimes just... swim away

I built three ballphin palaces: one at a time I don't specifically recall, and two in the evening. The first one's ballphin behaved as expected, but the ballphins from the two houses I built in the evening came out the following morning and just... left. I wasn't able to find them afterward, and they don't come home. They headed south by southwest from their homes; the only significance I can discern is that the ballphins who "donated" their fins were originally vaguely in that direction, but I didn't see any homeless ballphins there when I checked. They didn't head for the center of the map either.