Server Not Responding

Hello,

My husband and I will be playing the same game for hours and then disconnect get some food and when we go to get back online we will get the "Server is not responding error" when the second person tries to join the game. We are both on the same network sitting right next to each other yet one can get in and the other can not.