Being Unable to Click/Controller Disconnecting

Playing as Woodie with my partner being Wormwood, I use keyboard and they use controller.

Several times we both noticed that we would be unable to do actions. I wouldn't be able to click anything and their controller would disconnect.

For example, I used a moose idol and was unable to charge until i disconnected and reconnected. The prompt and icon would appear, but right clicking wouldn't do anything. I would also be unable to put wood on the fire, when I took it out of my inventory and hovered over the firepit the game would act as if there was nothing there. I couldn't drop things onto the ground either. The keyboard shortcuts for actions would still work, I just couldn't click anything.

We played for just 17 days and this happened several times. To fix it, I would disconnect from the server and my partner said they were minimizing then tabbing back into the game.