Mod API discussion thread

[nome]

As you may be aware there was recently an abuse of the permissiveness of the modding API where malicious code was spreading into other mods. You can find more information here.

While we were aware this would cause problems for certain mods the changes needed to be made immediately in order to prevent further spread of the malicious code and further utilization of the command and control functions built into that code.

With the immediate problem addressed we would now like to open a discussion with mod creators who have been limited by these changes so they can inform us of their needs and get a better idea of further changes that could be made.

Two changes we are looking at making very soon are:

• TheSim:QueryServer on either the client or server may now query to localhost or 127.0.0.1.
• io.open(filepath, "w") will work for filepaths that go into the "unsafedata" directory, and end with ".txt", ".tex", ".xml", ".png", or ".json".
• Example: io.open("unsafedata/mymodnamehere_data.txt", "w")

What we're wondering about is what mods are still broken even with the above two changes.

One use case we are aware of is paid entitlements in mods. We've been unable to endorse these because of the potential for player harm, but given the continuing popularity of this we are once again considering ways in which the user harms could be mitigated. As such we would appreciate hearing more about your use cases.

Please note that this is just an investigation at the moment and we cannot promise any changes we make will please everyone. But we love our modding community and hope we can enable the vast majority of use cases without harming player safety.

Please be sure to include as much detail as possible including at a minimum:

• a link to your mod
• which specific limited APIs you were using and why

[Mr.CrazyPotato]

Does that mean that we will be able to create files with io but just not .lua or .dll? If yes then I all in for this change.

I don't have a link for my mod, but I have modmain.lua. It basically creates copy of map like if it would be Tiled file. I can live with that it'll be the .txt file and I'll rename it because I need it for myself only.modmain.lua

I need it so I can share with my maps with other players and maybe import it into the Tiled and try editing it manually to see how it'll get. I use io here to create file which I'll write into, I use MODROOT to write into my mods folder. Mod creates table of all tiles and adds every object with the same skins, I'll later add containers.

Edited January 29, 2025 by Mr.CrazyPotato

[ADM]

Must say, it didn’t affect my mods well functioning directly, however my artist friends who work with me on various mods on which we mutually contribute lost a pretty important share of content we loved to distribute to players. I'm talking of how skins content was generally made by mods these days.

 

Since the changes, the commonly used Modded Skin API mod has stopped to work for "official" content to apply skins. This doesn’t just include the vanilla characters, items or structures, but also for clothes added from mod characters, custom collections, or mod prefabs that are generated by the game by inserting them in definition files and such cases.

 

To be fair it's always possible to make / use another mod skinning system but I only had words that this one specifically was acknowledged by Klei and their terms when it comes to make official skins, maybe because such skin files had to be .dynfied or because the sensible code of the API was obfuscated… I don't know but that's what its dev told me much back then so I always sticked to this. I don't want to be doing things that could be remotely illegal with your TOS. The other issue is that generally when multiple mods attempt to implement skins in their own ways it causes clashes between each others, completely crashing. So this one API made me happy thinking it unified a lot of the mod community under the same safe system to provide even cooler things.

 

I also speak of this because the dev of the mod has been taking an extensive break from modding so the future of it is very uncertain, I still really wish one day modded skins can be more-so supported by the game directly, but anything that allows me to do it, that is stable accros many mods adding them, and fine by Klei, is good to me.

Hoping our work can be back on the menu sometime soon 

Edited January 29, 2025 by ADM

[penguin0616]

My uses of QueryServer

1. One of my mods, Insight, allows users to opt-in to automatic crash reporting via QueryServer, which is used on both the server (for server crashes) and client (for client crashes).

This is my primary source (I would say almost 94%; the other 4% and 2% are from users properly reporting on Steam and from users on Discord, respectively) of information for resolving bugs and crash reports, as I've noticed a large majority of my user base is either unable or unwilling to manually retrieve their crash logs and provide them. The vast majority do not have the technical knowledge to be able to not only find their log files but also find the source of a crash and properly report it to the appropriate location. The cases in which they are able to do so is further hampered by Steam comments/discussions being an inadequate place to share this information, language and region barriers aside. 

The QueryServer restrictions would be very problematic for me in this case, as it effectively cuts off my source of information for issues and neuters my ability to quickly respond to issues. To help visualize how problematic:

Spoiler

What if the Bug Tracker was filled with these, with quite literally no information beyond the title? This is often my experience looking at the Steam page for my mod when there is an issue, with some instances having spanned multiple pages. Trying to get further information is often a painful process, especially as many comments are "fire and forget". And of course, this does not include the silent majority of users experiencing issues.

 

Some statistics:

Spoiler

Before the QueryServer restriction, I was receiving thousands of automated crash reports a month (batch for December 2024 was 3.4k). While my mod is not typically involved in the majority of them (maybe 10-12%), on a release cycle (be it a significant update for my mod or for the game), the amount of these reports that have my mod involved skyrockets (often to 30-40%). I have created backend tools for filtering and parsing the reports volume of reports I receive, and that enables me to quickly sort through them and uncover issues, many of which I would never have uncovered with my own testing. How many of those valid issues were reported manually by users? Less than 5%.

 

My uses of io

1. (Still works with above mentioned changes) One of my mods (not on the workshop) makes use of io to write to json and byte data to the data directory. This data stems from world/session data, miscellaneous data exports, etc. One of the key features is netvar debugging, where it writes the netvar states of the server and client to disk and performs comparisons to see where desyncs may be occurring.

2. (Impacted by above mentioned changes) Another one of my mods, Custom Vignettes, reads and writes to .json files included in the mod directory for editing the vignette data of the mod. It is specifically contained within the mod as it doesn't make sense to have it end up in the data directory, especially as the original source of data is shipped with the mod.
(A side note) This allows it to stay coupled with the mod, so the data doesn't linger if, say, the mod were to be uninstalled. The data in the hypothetical "unsafedata" would never get cleaned up and could be a problem for mods that write a lot of data and one day get uninstalled.

 

Edited January 29, 2025 by penguin0616

[HalfEnder776]

Writing lua files is important for using the scrapbook generator.....

[Asura_Dis]

I have been using io to create files that store customization settings and various in-game currencies (these currencies are not purchasable with real money) so that they can persist between sessions. Additionally, the mod collected statistics on the characters used in the game mode and sent crash logs via TheSim:QueryServer to a Discord server to avoid disturbing players during gameplay. Unfortunately, this is a private mod, and I cannot provide a link to it, as it uses a system similar to the one used during the Gorge and Forge events.

Additionally, a few months ago, I was working on a mod that needed to read .tex images from the worldtiles file, merge them into a single .tex file, and use that for VFX purposes.
I was also working on a private mod that ports houses from the Hamlet DLC. In this mod, we introduced a custom image format, which we then processed using io. The purpose was to read the image data bit by bit to determine its width and height, ensuring it could be correctly used in a vertex shader.

I have another mod called Ukrainizer, which is also private due to its Work in Progress status. In this mod, TheSim:QueryServer was used to fetch the Steam Workshop page and retrieve update information. This allowed us to avoid manually updating changelogs within the game, as the mod would automatically pull the latest updates from Steam.

Additionally, my colleague The Cunning Fox worked on a mod called Russian Language Pack. As we all know, players often don't update their mods unless they start causing crashes or other issues. Fox's mod checked the installed mod version against the version in the Git repository. If there was a discrepancy, it allowed the mod to load all updated strings dynamically, ensuring that the mod stayed up-to-date without bugs or other issues. Unfortunately, I can't say for certain how the loading of new strings worked. It could have been either a simple prompt asking the user to update the mod or an automatic update using built-in Steam-related functions.

[HalfEnder776]

I am using io.open(filepath, "w") by hooking into dsts scrapbook generator in order to automatically add extra scrapbookdata for new prefabs or automatically add changes to existing scrapbookdata for my mod. It would really suck if the scrapbook generator would now only be useable by developers who have access to the dev tools or we would have to manually copy the data in a lua file... I specifically put in the effort to hook into the dst one so that I would not need to keep it updated....

https://gitlab.com/IslandAdventures/ia-core/-/blob/IA-Core-Rewrite/scripts/ia_scrapbook_generator/debuggenerator.lua?ref_type=heads

We also use io.open(filepath, "w") to temporarily swap the waterfall texture files in map:Finalize(). This allows us to change the waterfall texture in game. This would no longer be possible with these changes. (I know this is an unintended hack, but this just seems like the best opportunity to ask for proper support to change the waterfall texture and shader...)

https://gitlab.com/IslandAdventures/IslandAdventuresHamlet/-/blob/IA-Ham---Rewrite/postinit/package/map.lua?ref_type=heads

Edited January 29, 2025 by HalfEnder776

[Cliffford W.]

2 hours ago, nome said:

As you may be aware there was recently an abuse of the permissiveness of the modding API where malicious code was spreading into other mods. You can find more information here.

While we were aware this would cause problems for certain mods the changes needed to be made immediately in order to prevent further spread of the malicious code and further utilization of the command and control functions built into that code.

With the immediate problem addressed we would now like to open a discussion with mod creators who have been limited by these changes so they can inform us of their needs and get a better idea of further changes that could be made.

Two changes we are looking at making very soon are:

• TheSim:QueryServer on either the client or server may now query to localhost or 127.0.0.1.
• io.open(filepath, "w") will work for filepaths that go into the "unsafedata" directory, and end with ".txt", ".tex", ".xml", ".png", or ".json".
• Example: io.open("unsafedata/mymodnamehere_data.txt", "w")

What we're wondering about is what mods are still broken even with the above two changes.

One use case we are aware of is paid entitlements in mods. We've been unable to endorse these because of the potential for player harm, but given the continuing popularity of this we are once again considering ways in which the user harms could be mitigated. As such we would appreciate hearing more about your use cases.

Please note that this is just an investigation at the moment and we cannot promise any changes we make will please everyone. But we love our modding community and hope we can enable the vast majority of use cases without harming player safety.

Please be sure to include as much detail as possible including at a minimum:

• a link to your mod
• which specific limited APIs you were using and why

I was used to sending mod crash logs into my discord server using a webhook

which starts with

https://discord.com/api/webhooks/

 

[amethystMushroom]

Discord Death Announcements also uses a Discord webhook. That's the whole mod; it just uses TheSim:QueryServer to call a Discord webhook whenever someone dies, to announce it in Discord.

[MAMjun]

The following is from machine translation:

MyMod:

 https://steamcommunity.com/sharedfiles/filedetails/?id=1638724235

1 I collect some server_logs when server or client is crashed and send this to my website.I use this to track the bug and .Most players will not provide logs, they will only say it crashed. i need this help me to I need this function to help me analyze the problem.

    using  in   /workshop-1638724235/main/logupload.lua

    used api: TheSim:QueryServer  and io.open(path,"w")

2 I have a hotfix system,it can fix some bug and don't need to restart the game. it get the updates by query my website.for eg,if i update my mods,all the game server use my mod need to restart the game .

This is very unfriendly for the game that is playing.

     using  in   /workshop-1638724235/scripts/soraupdate/

     used api: TheSim:QueryServer  and io.open(path,"w")

3 i have some skins ,players can unlock most of them by playing my mod,but i need to queryserver to record players is playing my mod.

     using  in   /workshop-1638724235/main/skin.lua

     used api: TheSim:QueryServer

 

 

 

I suggest:

   if a mod whats to use QueryServer ,it need to mark the domain name  needs to be accessed in modinfo.lua .example.:

                  QueryDomain = {"http://skin.flapi.cn","http://jh.flapi.cn"}

   if players enable the mods , The game will warn players. and when players join in a server who enable this mod,There will be the same warning.example:

                   This Mod will try to access some web site to provide additional functions, which may lead to some security issues, such as privacy leakage. If it is not necessary, please do not enable the MOD.

 

 

 

and if a mod want to io.open(path,"w") in self path.it can use  IO_Self = True in modinfo. if it mark this , game will warning players.

          IO_Self = True

 if the mod mark this, the dir of the mod will be add to a list,  io.open(path,"w") can only work in list.if this,mod can't to edit others mods who don't mark this.it will not bring risks to other mods.

 

 

 

for code :

    when the game loading the modinfo,read QueryDomain  and  IO_Self ,then add thes to a whitelist. after modinfo.lua  is loaded,the whitelist will can't be edited. when TheSim:QueryServer or io.open is called,it will check the path that needs to be accessed is on the whitelist, or it will be dropped.

 

 

 

Thanks you for you read.I hope the DST and MOD community are getting better and better!

logupload.lua skin.lua manager.lua

[Rickzzs]

I'd suggest you expose an API for modders to manually enable insecure features. And also for players, a toggle. The default behavior for io and queryserver is permission denied. If the player allows insecure action, it will be granted such permission. Otherwise, the game should ask the player whether to grant this mod this permission, granted or not, for once, or for all. The system can be seem in ANDROID system API and modern browser API.

To prevent easy crack, it should be c side code and only something like ThePermission:AskPermission() and ThePermission:HasPermission() is exposed to lua.

For example, the mod should first ask for a permission. In a system prompt (can be ugly, but isn't DST's UI entirely rendered in lua?) or in modsettingsoverride, the permission is granted.

Edited January 30, 2025 by Rickzzs

[Philip.]

I am also using TheSim:QueryServer not only for sending logs and reports to discord using the webhook REST API but also in a combination with long polling and a proxy server for fetching messages and commands from discord and sending them to the game.

In case if io.write, I was using it for caching stuff retrieved from the internet (discord emoji lists for generating fancy inventory embeds with icons for each item, public banlists, saving weights for my thief inventory detection model, ...) and I have also built a database for saving player data that supports multiple shards and therefore needs to have its files saved outside the mod's folder so it is accessible from any shard.

The reason why I try to use the Lua side as much as possible is because when you are using a hosting service like Nodecraft, they don't allow you to run some external scripts on their machines.

[GodIess]

The usual settings for any translation:
TheSim:QueryServer to get the po file from github and information about its version
io to check the version and if it is new, update the po file with the translation.

Also, from the additional functionality we used 
 

Spoiler

function loadULPCustomConfigFile()
	local strings = {}
	TheSim:GetPersistentString("../ULP_custom_config.txt", function(success, str)
		if success and str ~= nil and #str > 0 then
      local status, decode_str = pcall(function() return json.decode(str) end)
      if status and decode_str then
        strings = decode_str
      end
		else
			print("load custom config failed:", str, success)
			saveULPCustomFile({ ["STRINGS.UI.TEST.KEY"] = "Custom config, add more after the comma" })
		end
	end)

	return strings
end

function saveULPCustomFile(data)
	local tutorial = "/* Add new lines in the same format as the first option separating them with commas */\n\n"
	TheSim:SetPersistentString("../ULP_custom_config.txt", tutorial .. json.encode(data), false)
end

There was also a request to Crowdin to get information about the translation status(JSON), which was then displayed in the game itself
 

Spoiler

function TranslateProgress:_GetTranslatePercent(callback)
	TheSim:QueryServer("https://crowdin.com/backend/project/ulp/uk/get_files_tree", function(result, isSuccessful, resultCode)
		if resultCode ~= 200 or not isSuccessful or #result < 1 then
			print("[ULP] Get localized percentage: fail")
			callback(nil)
			return
		end

		local success, data = pcall(function()
			return json.decode(result)
		end)

		callback(data.files_tree)
	end, "GET")
end

 

Edited January 30, 2025 by GodIess

[Endeer]

Hello,

some time ago, I made a Chat Machine Translation mod: https://steamcommunity.com/sharedfiles/filedetails/?id=2402422659

Using free google API (or other API) to translate text of in-game chat, both receiving and sending.

	local query = "sl=" .. (sourceLanguage or "auto") .. "&tl=" .. targetLanguage .. "&dt=t&q=" .. urlencode(text)
	TheSim:QueryServer(
		"https://translate.googleapis.com/translate_a/single?client=gtx&" .. query,
        ...

This obviously no longer works. I don't really know the nature of the API abuse to propose any possible solutions which would work here.

Thanks for consideration.

Edited January 30, 2025 by Endeer
edited to finish the message

[Peasant]

I totally approve that banning all the paid entitlements in mods. If people want to support the modders for their hard work they should be allowed, but none should be gaining extra mod contents because he sponsered the modder.

[Notecja]

I am artist working with ADM, and we were using Modded Skin API for character skins, vigettes and skins for offcial items in theme of our mods within the rules of Modded Skin API (.dyn, free to use). Without working API some of my work is going to be just forever gone, and others might be not as stable and compatible as that one. It was good to unify things, so there was less problems between the mods.

[Love_MapleSyrup]

I have reviewed the latest update. Are you not planning to make any changes to TheSim:QueryServer? Maybe you're not ready yet? Please give a clear answer.

Otherwise, the mod for information exchange between my live streaming room and the game will have to be declared over.

[BB Goat]

My translation mod https://steamcommunity.com/sharedfiles/filedetails/?id=2941527805 uses TheSim:QueryServer to access my server at server.bbgoat.cn

I receive new translation requests from time to time, but every time I upload a new translation update to the Steam Workshop, players complain that my mod updates too frequently. So, I use TheSim:QueryServer to hot-update the translation files.

Edited February 6, 2025 by BB Goat

[BB Goat]

50 minutes ago, BB Goat said:

我的翻译 Mod https://steamcommunity.com/sharedfiles/filedetails/?id=2941527805 使用 TheSim：QueryServer 来访问我在 server.bbgoat.cn 的服务器

我不时收到新的翻译请求，但每次我向 Steam 创意工坊上传新的翻译更新时，玩家都会抱怨我的 Mod 更新太频繁。因此，我使用 TheSim：QueryServer 来热更新翻译文件。

The update requires write permission using io.open, and the file needs to be written in the mod's own directory. :/

[-LukaS-]

Hope it's not too late now but would it be possible to add .ksh as an acceptable extension for io.open?

Just a few weeks before this change I made a shader API that helps with creating more complex shaders but it heavily relies on io.open and creating .ksh files during runtime. They are required to be created during runtime because they use a uniform variable as an ID and depending on which shaders are used the IDs can change.

This is the mod:

https://steamcommunity.com/sharedfiles/filedetails/?id=3345578390

[JesseB_Klei]

3 hours ago, -LukaS- said:

Hope it's not too late now but would it be possible to add .ksh as an acceptable extension for io.open?

This is an ongoing subject and the file extension will be allowed in the next hotfix.

[RexySeven]

My mod (https://steamcommunity.com/sharedfiles/filedetails/?id=2978133982) is using Modded Skin API to add character skins and vignettes. 
Having official way of making modded skins or vignettes would be much appreciated! Using third party API with uncertain future support is very unreliable and harming in the long term.