SendModRPCToClient not working as intended

I made a mod named Craftable Wormholes, where I added wormhole icon support in the last update. I use SendModRPCToClient to make the client delete the icons. This works if I use newly created wormholes, but if I try to do it when an original wormhole that was created with the world is connected to a new wormhole, nothing happens. I get no error in the client or server log, just nothing. If I only do it with the new wormholes, it works without problems.

I added prints to check if the RPC is indeed sent and they appear in the log.

The code in question:

local function onhammered(inst, worker)
    GLOBAL.TheWorld:PushEvent("wormhole_destroyed",{wormhole = inst})
    if GetModConfigData("ENABLED") then
    	if inst.components.teleporter.targetTeleporter ~= nil then
    		local userids = {}
			for i, player in ipairs(GLOBAL.AllPlayers) do
				table.insert(userids, player.userid)
			end
			SendModRPCToClient(GetClientModRPC("Wormhole_Crafter", "RemoveWormhole"),userids,inst,inst.components.teleporter.targetTeleporter)
    	end
    end
    inst:DoTaskInTime(0.3, function() inst:Remove() end)
end

local function RemoveWormhole(wormhole_removed,wormhole_still_here)
	local hole_removed = {inst=wormhole_removed,pos = wormhole_removed:GetPosition()}
	local hole_still_here = {inst=wormhole_still_here,pos = wormhole_still_here:GetPosition()}
	RemoveWormholePair(hole_removed,hole_still_here)
end

AddClientModRPCHandler("Wormhole_Crafter", "RemoveWormhole", RemoveWormhole)

The SendModRPCToClient function is called and goes through the assert which means that the error happens somewhere in TheNet:SendModRPCToClient, as I don't get a response in HandleClientModRPC.

The mod in question: https://steamcommunity.com/sharedfiles/filedetails/?id=2524038682