Getting major disconnects when launching server/porting wormholes or leaving caves, it works for a few seconds before booting me out of the server

It started to happen again, I decided to restart the game and start shooting a video of it: 

 

Update: I'm also getting kicked out of the game when I rollback, I have a friend who said rolling back causes him laggy and got him kicked out when he rolls back.

I rolled back a few 2-3x this week due to beefalo death, etc.... it kicked me out of the game when I rollback.

It crashes when I rollback, when I come back up from the caves, or when I simply start the game 

PS4.zip