Caves progress not synced with overworld progress

My caves progress sometimes gets desynced with overworld progress. I have lost 3 worlds now due to this bug. I play DST solo, saved to Steam Cloud.

The first time this happened, I went down to the caves to gather lightbulbs. When I spawned in to the caves, I noticed that my inventory was different compared to what I had in the overworld immediately before coming down to the caves. The day counter was pushed back a couple of days and character progress (inventory, days survived) was reverted to reflect the day counter. I considered this a loss and deleted the world to start a new one.

I was more attentive the second time when it happened again. I was going down to the caves close to when the night > day cycle was going to change. Upon entering the caves, my inventory changed to when I last left the caves and the day counter was pushed back a couple of days (the same day count as when I left the caves the last time). It also seems that 2 overworld cave entrances brought me down to the same cave. Frustrated, I deleted the world to start over again.

Today, I have a video of what happened (available on request). I went down to the caves with a good amount of time before night and when I spawned in, I was out of bounds and had to use the /rescue command. My inventory was different to what I had in the overworld. The day counter decreased by 10 days (which is approximately when I last left the caves). My character progress was reverted to reflect the day count - I had killed dfly prior to entering the caves and after my day count was reverted, I had lost the blueprint for the Lazy Furnace.

Before entering caves. Day 155. 3 segments before night,

After entering caves. Out of bounds. "day 155" (but when I pressed Tab I only survived 145 days). Inventory is different. hunger/sanity/health is different.

Mods: (S) Server mod, rest are client
(S) Domestication Fixed
(S) Epic Healthbar
(S) Lazy Furnace
ActionQueue Reborn
Combined Status
Geometric Drop
Geometric Placement
Minimap HUD
Quickdrop
SmartCrockPot
Snapping Tills