Why does this keeps happening? I need help.

[Kommu]

I use to play DST with my friends a lot, and my Wi-Fi connection is pretty stable so i'm usually picked as the host, but something odd happens whenever i'm it.
When i host, i notice that whenever i drag my mouse above an item in my inventory or one of my inventory slots, it adds a "frame" of lag everytime i do it, and by day 20 - 30 i have to restart the server because otherwise i just can't put my mouse over my inventory, because if i do it drops my FPS to 5 and the server starts working embarasingly wrong, and i'm forced to use the numbers of my Keyboard.
I have no clue why does this happens, but it happens A LOT,  and it's disturbing as hell, and as far i've seen, this doesn't occurs to nobody of my group, i'm literally the only one who has this problem. 
This only happens when i'm the host, it doesn't matter if i'm playing vanilla, low graphics/netbook mode, with mods, etc, IT-ALWAYS-HAPPENS, even when i'm playing alone.
An important note to say is that whenever i'm not the host, this still occurs, but at a WAAAAAY less worse level

Anyone has any idea of what could be the source of the problem here and how can i fix this? I'm pretty sure my PC is better than what DST asks to run properly, yet this stills happens.
This never happened to me on regular DS, and neither the first days of my DST experiences.
Thanks a lot before hand to whoever reads this!

EDIT: If it helps trying to find the problem, 2 of the friends i play with have worse PC than mine, yet their game works 100% better than mine. I've already trying rebooting my PC to 0 or cleaning it up entirelly, yet this still happens.

[ArubaroBeefalo]

It sounds like a performance problem with a mode like display food values or show me

Do you have some kind of mod that shows stats from items?

[Kommu]

Yeah, but as i say, this happens ALWAYS, doesn't matter if i have mods or i'm playing vanilla.
I've also tryied disabling entirelly my mods list, including the Client ones, but this still happens.