Friend could not join server

I have a friends only server; someone on my friend list could not join the server. It perpetually stayed on loading screen, except when I paused the game - his game would stop the music and it'd say the server was paused.