Please Make Modding Sustainable

[asquared31415]

This list is compiled from talking to several other modders and discusses several of the things that make it hard to create reliable mods for ONI.  We love this game, and love to create new things, but the state of the game and the way that things change make it very hard.  If any developers could address our concerns, we would be grateful.

Compatibility Issues
The recent addition of `int HorizontalGroupID()` to `ISidescreenButtonControl` is a breaking change.  Mods that tried to add a button to a side screen will fail to load with an error related to "cannot build vtable".  In the Hot Shots update, `void SetButtonTextOverride(ButtonMenuTextOverride textOverride)` was also added to this interface, breaking mods then too.  Modifying any public interface will break mods because mods may try to call a method that no longer exists or be missing the methods needed for the interface.

Renaming things with no real benefit is also fairly common and causes issues for mods.  Public members should never be renamed, but even private things are frequently used by mods, and it's frustrating when a random drive by change changes something like `private_Ping` to `Internal_Ping` (in this case the method is `private`, not `internal` so it's even a downgrade in terms of name) and the method is only used in a few places, but now mods break for these useless sorts of changes.  Mods often call private methods or use methods or fields to find a location in a method to patch, so stability of names is important.

Changing a method signature is always a breaking change.  Even if the signature has defaults for a new parameter, compiled code requires a specific full signature and calls it with the defaults.  A change along the lines of `SomeMethod(int x)` to `SomeMethod(int x, int y = 0)` breaks compiled references, since they expect a `SomeMethod(int)`, not a `SomeMethod(int, int)`.  Adding a new overload for the method with the new argument, then having both methods call shared logic is much more compatible, if changes to a signature are required.  Additionally, changing default arguments to methods can cause unexpected behavior because mods keep the old default from when they were compiled.  A mod compiled against `SomeMethod(int x, int y = 0)` will not use the new default if the method is changed to `SomeMethod(int x, int y = 1)`.

Adding `enum` variants anywhere besides the end of an `enum`, removing `enum` variants, or changing the order of enum variants will cause compiled mods to function improperly.  Compiled code just uses an integer value for `enum`s, and if the backing value changes because variants are changed, the compiled code will have the wrong variant when the game interprets this integer.  `enums` are also a nightmare for mods in general because they are exceptionally hard to extend, the extensions are prone to breaking, and multiple mods can end up trampling each other.

Removing or renaming serialized data is a breaking change.  Mods rely on serialized data on base game components to change the way that those components behave and persist that data across loads.  An example of this was the changes to `MinionIdentity` that stopped saving the body data for each dupe, instead looking it up by the `Personality`.  This broke a mod that introduced a very large variety of new duplicant styles, called Bio-Inks, and even worse, that data was lost forever as soon as a player loaded the save in the new version, so the data could not be migrated.  Leaving serialized data alone, especially with an `[Obsolete]` attribute, gives modders time to rescue users' data before it is permanently gone.

Because removing, renaming, or changing the signature of any class, field, method, or other member has the potential to break any mod, we would appreciate if unused or legacy code could remain in the game, possibly with an `[Obsolete]` attribute.

Obsoleting
The `[Obsolete]` attribute should be added before making breaking changes, to give modders more time to see a change, instead of having to search for similar game code to see what changed.  An example of this being done well was when there were changes to the `PlanScreen` and that affects how mods should add to it.  The `PlanScreen.PlanInfo.data` field is annotated with `[Obsolete("Modders: Use ModUtil.AddBuildingToPlanScreen")]` to inform modders of the proper API.

For example, the removal of the `TagBits` class and its methods entirely was likely not justified.  Even if the class is never used, or the methods never called, keeping the class in the game allows mods to continue to work as much as possible.  Mods tend to react much better to never being run, where as removal of a class always causes a crash, even if the mod didn't necessarily need to run the code.  Ideally this class would have all of its methods left intact and the `[Obsolete]` attribute placed on the class and all methods and fields, with a method directing modders to the new systems that replaced it.

In general, if there are concerns about needing to maintain old code, it could be removed, but at least one full release with code marked as obsolete prevents unexpected compatibility issues.  Many modders don't test against the public testing branch because it is so unstable and code changes all the time.  When we know the expected release date, sometimes we can do last minute tests, but even then, modders normally don't release updates for testing versions.  There will inherently be a period of time that a breaking change is made and no mod update has happened.  Having a release cycle with code marked as `[Obsolete]` but not removed eases the transition for modders and users across changes and reduces the number of "broken pls fix" (normally with no helpful information) comments that modders receive when game code suddenly goes missing.

Communication with Modders
Good communication with modders makes them much more receptive to change and gives them a chance to provide feedback, particularly for compatibility concerns.  In the past there have been a few sentences in patch notes that generally outline the high level changes, but more specific technical changes would help greatly.  Notes like "added two new methods to class `SomeClass`", "made class `OtherClass` obsolete, use class `NewClass` instead", "class `NewFeature` was added and is the preferred method to do <some action>", or "class `ReallyBigClass` was significantly changed, with several methods and fields added and removed" are very helpful for modders.  With specific technical changes, a modder can easily search through their code to see if they interact with that code.  Technical communication can also be achieved in several cases with the `[Obsolete]` attribute, as mentioned above.

Communication back from modders to ONI developers is also extremely limited.  The Klei forums seem to be the best place to interact with the team, but posts often seem to go unnoticed and there's no good place to put technical feedback or requests (like this post).  There are no ongoing discussions, communication from modders to developers feels very one-way via forum posts.  While we appreciate when Klei responds, we have no way to know if our communication was received, and slow communication or posts getting buried can stall mod development possibly indefinitely due to unclear rules or critical bugs that mods can't work around.  One example of these posts that need a response to continue development regards the new Blueprints skins system and what is or is not acceptable to add to that system.  Without knowing whether mods are allowed to add custom art to the blueprints system, modders cannot safely release their mods that have new art.  What is the best way to contact developers, and is there anything that can be done to allow for more consistent communication?

Difficulty Accessing Things
Even with the power of reflection on our side, it is very difficult to access a `private struct`, and even harder to access a generic type that uses one, like `List<SomePrivateStruct>`.  For example, `ModsScreen.displayedMods` uses a private struct `ModsScreen.DisplayedMod` but mods need to access it to modify their mod entry, such as adding buttons to the mod menu for settings.  Even private nested classes are preferable to private structs, since they can be used as `object`, but it's still very difficult.  Additionally, large structs have performance issues when passed around, since they need to be copied to the stack everywhere.  If something must be a struct, for example for communicating with the native libraries, it should be public.

Several private structs that modders have wanted to use include:

• `ModsScreen.DisplayedMod` for adding settings buttons to the mod menu
• `Game.SpawnPoolData` to add custom fx
  • `Game.SpawnRotationConfig` as part of the above struct
  • `Game.SpawnRotationData` as part of the above struct
• `UnstableGroundManager.EffectInfo` to add particles to new elements that fall like sand

We would appreciate if these could be made public, and private structs made public where possible.

Dev Tool Improvements
Currently, the method `DevToolManager::RegisterAdditionalDevToolsByReflection` does not collect any additional dev tools.  It runs well before mods load, so it can't fetch tools from mods, and all dev tools are currently manually registered in the constructor for `DevToolManager`.  From my own experience, and talking to other modders, we do not want this method to run on modded dev tools.  It does not provide as much control as we would like over the dev tools.

The best way for mods to add a dev tool is currently to reflect for the method `DevToolManager::RegisterDevTool`, make it generic for their tool, and call it.  We would like this method to become public.  This method allows a custom path for the dev tool, instead of forcing us into using the very cluttered `Debuggers` tab.  Several modders have already informally started a convention to use `Mods/` as the root for their dev tools, which is only possible by using this private method.

The dev tools also have several bugs that make using them more difficult, explained in bug reports linked below.

Workshop Issues
The version of the Steam Workshop that ONI uses for user content is an outdated version of the workshop, using old unsupported APIs and it fails to consistently download updates due to a variety of bugs from both Steam's CDN and ONI's code not properly downloading mods, missing and deleting files, and a variety of other failures.  In extreme cases I have seen mods that are 8 months out of date, when there were several newer versions available across that time span.

From Peter Han, issues that ONI could fix are:

• Leaked zipfile handles are locking the mod zips (and stopping Steam from updating mods)
• A race condition in the update downloader can cause some downloads to be aborted
• Bugs where the content available / version / yamls are being pulled from the outdated Steam files rather than the local correct files

Because we cannot ensure that users get updates to our mods in a timely manner, it is much harder to justify putting out small fixes (such as recompiling the mod since a new default argument was addded) and the general development process would be much easier.  Additionally inconsistent updates make it harder to have mods interact with each other.  Two authors can't just perform an update on both of their mods at the same time, since many users will not get both updates at the same time.

The workshop also has some file size and file count limitations.  Because mod authors would like to support as many game versions as possible, they use the archived versions system and try to keep them around forever.  But that means that every time that there is a breaking change (which is very often) another entire copy of the mod has to be added to the mod folder.  For mods that have lots of assets like art, this can mean that the limits are prohibitively small.  One user reported that they were not able to keep any archived versions, even with heavy compression of everything that could be compressed.  At least one in development mod is above the workshop limits with no archived versions, meaning the mod cannot be released on the current workshop at all.

Cross-mod Compatibility
Currently, mods cannot properly depend on another mod.  If mod A references mod B, but mod B is loaded first, or mod B is not present, most of the time mod A will crash.  This inherently cannot be fixed, but in certain situations, the error can be caused by a type trying to inherit from another type, and in that case, `assembly.GetTypes()` (used to find a `UserMod2`) throws a `ReflectionTypeLoadException`.  Unfortunately this exception does not have the useful info in the default message, it only says `System.Reflection.ReflectionTypeLoadException: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.`  It would be more useful if `KMod.DLLLoader.LoadDLLs` had a special case to handle `ReflectionTypeLoadException`, access its `LoaderExceptions` property, and print out each of those exceptions.

The proper way for mods to interact is to interact with each other cooperatively, like adding additional features if another mod is present.  While `UserMod2.OnAllModsLoaded` provides a good way to determine if another mod exists, communication is difficult and often involves reflecting for methods and fields anyway.  We would like a `Dictionary<string, object>` field that is easily accessible on `LoadedModData` or `Mod` that mods can use to communicate with each other.  This would allow mods to access documented values that other mods provide without needing to reflect for fields or methods.  An accessor on `UserMod2` could also be convenient for mods to insert their data without needing to go through the `mod` field.

Unity Player Settings

Managed Code Stripping appears to currently be enabled in some form, there are missing Unity components and several parts of the .NET Runtime are missing.  Many things that Klei don't use are gone from the build and completely inaccessible.  We would like to have access to these parts of the Mono Runtime and Unity components.  The size difference in managed assemblies should be negligible compared to the multiple GB of assets and other data, and provide much more utility to modders.

Enabling Incremental GC may improve the performance of the game by reducing the intensity of lag spikes created by garbage collection.

Compatibility Bugs in Whatta Blast
- Modifying `ISideScreenButtonControl` breaks compiled mods Bug Report
- Removing `TagBits` broke a few mods

Bugs that impede mod dev:
- The Dev Tools cannot be accessed on some alternate keyboard layouts Bug Report
- The Dev Tools crash when `,` is pressed when not hovering over a UI Bug Report
 - The linux native `cimgui.so` library is missing the `SetKAssertCB` used by the dev tools, so the game crashes to desktop when opening dev tools on linux Bug Report
 - Custom element `Substance`s have their `nameTag` overwritten with their *display name* but it should be from the `Element`s `id` so that the element doesn't crash when you dig it Bug Report

Summary of specific change requests
Breaks Mods/Mods can't do:

• Too late for this, it made it in to a public release  (Compatibility) Remove the newly added `int HorizontalGroupID()` to `ISidescreenButtonControl`
• (Workshop) Clean up mod downloader zip file handles to let Steam update mods more consistently
• (Workshop) Raise the file quantity and size limits
• (Mod Loader) Add handler for `ReflectionTypeLoadException` in `KMod.DLLLoader.LoadDLLs` to print out each exception in `LoaderExceptions`
• (Unity) Disable Managed Code Stripping to let modders access the whole .NET runtime and more Unity components

Annoying Bugs/Limitations:

• (Dev Tools) Fix crash on Linux when opening dev tools
• (Dev Tools) Fix crash when pressing `,` when not hovering over a UI
• (Dev Tools) Make dev tool open keybind configurable or work with other layouts
• (Access) Make `ModsScreen.DisplayedMod` public
• (Access) Make `Game.SpawnPoolData` public
• (Access) Make `Game.SpawnRotationConfig` public
• (Access) Make `Game.SpawnRotationData` public
• (Access) Make `UnstableGroundManager.EffectInfo` public
• (Elements) Make `Substance.nameTag` always based on the `Element`'s `id`
• (Workshop) Collect mod.yaml and mod_info.yaml info from the locally unzipped mod files instead of the steam download to show the right version

Feature Requests:

• (Mod Loader) Add `Dictionary<string, object>` field to `LoadedModData` or `Mod`
• (Dev Tools) Make `DevToolManager.RegisterDevTool` public

[Peter Han]

Another point to add - renaming method parameters can break mods as well. Many mods patch methods and use Harmony to obtain the parameter values, but the names of the parameters are used to bind the patch. If parameters are renamed, any patches to that method which use them will fail to apply. For example, Whatta Blast renamed a parameter from "dc" to "dcs" in world gen, which broke a few mods unnecessarily.

[RomenH]

On 3/29/2023 at 6:18 PM, asquared31415 said:

While we appreciate when Klei responds, we have no way to know if our communication was received, and slow communication or posts getting buried can stall mod development possibly indefinitely

I really hope that somebody at Klei will at least acknowledge this thread. It is disheartening as a modder to see my bug own reports going unnoticed and it has put some of my mod ideas on hiatus.

[SGT_Imalas]

On 4/7/2023 at 3:19 AM, RomenH said:

It is disheartening as a modder to see my bug own reports going unnoticed and it has put some of my mod ideas on hiatus.

^ prematurely put some mod ideas on hold indefinitely

[asquared31415]

The current beta update breaks in several of the ways outlined in this post. Lots of `KAnim` things moved from a class to a namespace, which breaks every compiled mod that plays an animation not using the default systems. A few methods in `Components` changed (and it's unclear why), so compiled mods that use that break.

Other authors have filed bug reports for both of these specific cases, but the fact that this continues to happen is very unfortunate.

[pether]

I would love to see some answer or reaction from Klei on this topic, it's been 3 months and I'm not even sure if they saw this post... And the topic feels pretty important for anyone making or using mods... I know Klei isn't really responsible for keeping mods alive, but it would be nice, especially since many people in the community seem to enjoy playing with them