Can't connect to certain friends

I have played other games with my friends and every time he and a certain few try to play dst in Xbox one we can't connect in a server. We keep getting a message saying you where disconnected...we don't know why. Mind you we play other games together. We tried closing the game out and joining, joining from the server list and invites. Tired to make the server public but still nothing and we have good connection so how can I fix this?