Second person to load the world has a different world loading with playing LAN

Either when my brother hosts the game or I host the game the other person who joins the game loads into a world with a different base map. The host's map makes sense but the person who joins the game has a terrain that doesn't make sense but with all the same objects. The joining person finds objects on the water, barriers that are invisible (the ocean for the host) and can not cross the water seen in his own terrain. We have tried restarting multiple times and unistalling over and over.