Ballphins don't leave player made houses

Once you build a ballphin palace, the ballphin will spawn outside. At night it will go to sleep, and never come back out, keeping the lights off in the house. The only way to force them out is to hammer the house.