Keeping data in sync in between shards

[Hamurlik]

Hello. I have this component, PostPostInit. Its functionality doesn't really matter, but basically it holds some functions that are then used in AddPrefabPostInitAny, so I can effectively change prefab post init while the world is already loaded. I use it for testing purposes

 

../modmain.lua

Spoiler

modimport("imports/postpostinit_import")

../imports/postpostinit_import.lua

Spoiler

local env = env
GLOBAL.setfenv(1, GLOBAL)

env.AddPrefabPostInitAny(function(inst)
	if inst == nil or not inst:IsValid() or TheWorld == nil or TheWorld.ismastersim ~= true then
		return
	end

	--Make sure we load the very very last
	inst:DoTaskInTime(0.1, function()
		if PostPostInit == nil then
			return
		end

		for index, init in ipairs(PostPostInit.inits) do
			if init.test(inst, init) then
				init.init(inst)
			end
		end
	end)
end)

env.AddPrefabPostInit("world", function(inst)
	if inst.ismastersim then
		inst:AddComponent("postpostinit")

		--For easy access. Remember it's only on mastersim
		global("PostPostInit")
		PostPostInit = inst.components.postpostinit
	end
end)

../scripts/components/postpostinit.lua

Spoiler

--[[
	How to use:
		
		local function TurnRed(inst)
			if inst.AnimState then
				inst.AnimState:SetMultColour(1,0,0,1)
			end
		end

		local function UnTurnRed(inst)
			if inst.AnimState then
				inst.AnimState:SetMultColour(1,1,1,1)
			end
		end

		local id = PostPostInit:AddInit({init = TurnRed, deinit = UnTurnRed, test = "prefab", prefab = "axe"}, 0, true)
		PostPostInit:RemoveInit(id)
]]

local preset = {
	prefab = function(inst, init)
		return init.prefab == inst.prefab
	end,
	tag = function(inst, init)
		return inst:HasTag(init.tag)
	end,
	any = function(inst, init)
		return true
	end,
	player = function(inst, init)
		return inst:HasTag("player")
	end,
}

local PostPostInit = Class(function(self, inst)
	self.inst = inst

	if self.inst ~= TheWorld then
		print("CR3: Tried to assign PostPostInit to an instance that isn't TheWorld.")
		return
	end

	self.inits = {}
	self.id = 0

	self.shard_sync = true
end)

function PostPostInit:DebugPrint()
	for i, v in ipairs(self.inits) do
		print("self.inits", i, v)
		for k, v in pairs(v) do
			print(i, k, v)
		end
	end
end

function PostPostInit:GiveId()
	self.id = self.id + 1
	return self.id
end

function PostPostInit:FindInit(id, find_index)
	for i, v in ipairs(self.inits) do
		if v.id == id then
			return find_index and i or v
		end
	end

	return nil
end

function PostPostInit:AddInit(init_data, priority, apply_to_existing)
	local id = self:GiveId()

	local init = init_data

	init.priority = priority
	init.id = id
	init.should_apply = apply_to_existing
	init.should_remove = false

	if type(init.test) == "string" then
		init.test = preset[init.test]
	end

	table.insert(self.inits, init)

	self:RefreshInits()

	return id
end

function PostPostInit:RemoveInit(id)
	self:FindInit(id).should_remove = true

	self:RefreshInits()
end

function PostPostInit:SortInits()
	table.sort(self.inits, function(a, b)
		return a.priority > b.priority --Highest priority loads first, and may get overwritten by lower priority.
	end)
end

local function apply_to_ents(init, removing)
	for _, inst in pairs(Ents) do
		if inst ~= nil and inst:IsValid() then
			if init.test(inst, init) then
				if not init.should_remove and init.should_apply then
					if init.existing then
						init.existing(inst)
					else
						init.init(inst)
					end

					init.should_apply = false
				else
					if init.deinit then
						init.deinit(inst)
					end
				end
			end
		end
	end
end

function PostPostInit:RefreshInits(sent_from_shard)
	-- if not sent_from_shard then
	-- 	SendModRPCToShard(GetShardModRPC("CR3", "postpostinit_sync"), nil, self)
	-- end

	self:SortInits()

	for index, init in ipairs(self.inits) do
		if init.should_apply or init.should_remove then
			apply_to_ents(init)
		end

		if init.should_remove then
			table.remove(self.inits, index)
		end
	end
end

return PostPostInit

 

The component is added to TheWorld, as you can see in postpostinit_import. It works perfectly fine and as intended on singleplayer worlds without caves, where the player is the master sim. I have not done much testing outside those conditions.

The problem is that in servers with caves, overworld and caves have separate instances of the component, and I need them to be the same on both shards. My first thought was to add shard rpcs, however they cannot take functions as arguments, neither can DataDumper or json.encode. Functions are necessary for how this component works; I must be able to supply a completely unique function through the console at any time, and have it synced to caves as well.

I'm stuck. How would you add such a thing?

Edited September 25, 2023 by Hamurlik

[Hamurlik]

Here is a failed attempt with a shard rpc:

modmain.lua

AddShardModRPCHandler("CR3", "postpostinit_sync", function(sender_shard_id, dumped_data)
	if PostPostInit == nil or TheShard:GetShardId() == sender_shard_id then
		return
	end

	local data = loadstring(dumped_data)()

	for k, v in pairs(data) do
		print("data layer 1", k, v)
		if type(v) == "table" then
			for i, v in ipairs(v) do
				print("data layer 2", i, v)
				if type(v) == "table" then
					for k, v in pairs(v) do
						print("data layer 3", k, v)
					end
				end
			end
		end
	end

	PostPostInit.inits = data.inits
	PostPostInit.id = data.id
	PostPostInit:RefreshInits(true)
end)

 

postpostinit.lua

function PostPostInit:RefreshInits(sent_from_shard)
	if not sent_from_shard then
		SendModRPCToShard(
			GetShardModRPC("CR3", "postpostinit_sync"),
			nil,
			DataDumper({ inits = self.inits, id = self.id }, nil, true)
		)
	end

	self:SortInits()

	for index, init in ipairs(self.inits) do
		if init.should_apply or init.should_remove then
			apply_to_ents(init)
		end

		if init.should_remove then
			table.remove(self.inits, index)
		end
	end
end

 

And the error log that follows:

[00:01:17]: data layer 1	id	1	
[00:01:17]: data layer 1	inits	table: 000000000BA0A990	
[00:01:17]: data layer 2	1	table: 000000000BA0AA30	
[00:01:17]: data layer 3	prefab	axe	
[00:01:17]: data layer 3	should_apply	true	
[00:01:17]: data layer 3	priority	1	
[00:01:17]: data layer 3	should_remove	false	
[00:01:17]: data layer 3	id	1	
[00:01:17]: [string "../mods/Creative_Mode_3/scripts/components/..."]:109: attempt to call field 'test' (a nil value)

As you can see, DataDumper got everything but the functions.

Rpcs do not accept functions as a valid argument data type either, so not using DataDumper won't fix it.

[_zwb]

On 9/24/2023 at 10:26 PM, Hamurlik said:

however they cannot take functions as arguments,

Maybe you can define those functions as methods of the component you use?

[Hamurlik]

3 hours ago, _zwb said:

Maybe you can define those functions as methods of the component you use?

That wouldn't really do what I want it to do - see, the goal is to use the component from the console and be able to supply any function to it while the game is already running, not any predefined function already stored in the component