[Tutorial] Function hooking and you

[CarlZalph]

In LUA, it is a very important concept to understand that everything is a variable and all variables may be edited in runtime.  This includes functions.

With modding other peoples' LUA files, like Klei's basegame code, you may find yourself wanting to run your code before or after the original function's call.  You might want to edit the original function's arguments, or change the return results from the function.

 

In LUA it is almost trivial to hook a function, but to do it well it can be a bit indirect and uses a few concepts new users to LUA might not know of.

 

Let's say you see some table having a function as part of its 'object':

SomeObject = {}
function SomeObject:SomeFunction(arg1, arg2)
    print("SomeFunction", arg1, arg2)
    return arg1 + arg2
end

This is functionally equivalent to:

SomeObject = {
    SomeFunction = function(self, arg1, arg2)
        print("SomeFunction", arg1, arg2)
        return arg1 + arg2
    end,
}

The object:function(args) syntax is syntactic sugar so you do not need to use object.function(object, args), the colon denotes the "use object as the first argument to a function".

This is important for function hooking so you can properly align your arguments up with what the base function is actually doing.

 

For the following example printouts I will be issuing the following to the function each time:

print(SomeObject:SomeFunction(1, 2, 3, 4, 5, 6))

Note the extra arguments that the base function isn't using.  This is to simulate if the function you're hooking changes in the future, like if you're hooking a Klei function and Klei edits it to have more parameters.

 

 

So let's start with a basic hook that runs some of your custom code before and after the target function:

local SomeFunction_old = SomeObject.SomeFunction
SomeObject.SomeFunction = function(self, arg1, arg2)
    print("PRE hook!")
    local ret = SomeFunction_old(self, arg1, arg2)
    print("POST hook!")
    return ret
end

Running this and invoking the function outputs:

PRE hook!
SomeFunction	1	2
POST hook!
3

Which is a good thing, now you have some custom code running before and after the base function being called.

In the pre hook area you can modify arguments on the fly.  In the post hook area you can modify return results.

local SomeFunction_old = SomeObject.SomeFunction
SomeObject.SomeFunction = function(self, arg1, arg2)
    print("PRE hook 1!", arg1)
    arg1 = arg1 * 10
    print("PRE hook 2!", arg1)
    local ret = SomeFunction_old(self, arg1, arg2)
    print("POST hook 1!", ret)
    ret = ret + 5
    print("POST hook 2!", ret)
    return ret
end

Prints:

PRE hook 1!	1
PRE hook 2!	10
SomeFunction	10	2
POST hook 1!	12
POST hook 2!	17
17

So now the base function is being modified for its arguments and return values.  Spiffy.

 

But there's an issue with how the hook is being created.  It doesn't take into consideration for future updates to the function.

Let's say Klei changed the base function to:

SomeObject = {}
function SomeObject:SomeFunction(arg1, arg2, arg3, arg4)
    print("SomeFunction", arg1, arg2, arg3, arg4)
    return arg1 + arg2 + arg3, arg4
end

Prints:

SomeFunction	1	2	3	4
6	4

But now it also returns two values instead of just one.

 

If we use the same hook above then the printout looks like:

PRE hook 1!	1
PRE hook 2!	10
SomeFunction	10	2	nil	nil
input:4: attempt to perform arithmetic on a nil value (local 'arg3')

Well that crashed.  Thanks, Klei, now you have to go in and fix your mod!  More work!

But it doesn't have to be.  Not quite.

 

So the base function uses another parameter in its calculations that you're not taking into account for.

You could make your function hook have arg3, arg4, arg5, arg6, arg7, arg8, arg9, ... up to arg N in your function prototypes.  But that's a lot of extra typing.

LUA provides the nice variable argument parameter "..." to use to denote "There could be N arguments here, I don't know, but use them if they exist."

 

So to change the function hook, add it in to the arguments list:

local SomeFunction_old = SomeObject.SomeFunction
SomeObject.SomeFunction = function(self, arg1, arg2, ...)
    print("PRE hook 1!", arg1)
    arg1 = arg1 * 10
    print("PRE hook 2!", arg1)
    local ret = SomeFunction_old(self, arg1, arg2, ...)
    print("POST hook 1!", ret)
    ret = ret + 5
    print("POST hook 2!", ret)
    return ret
end

Prints:

PRE hook 1!	1
PRE hook 2!	10
SomeFunction	10	2	3	4
POST hook 1!	15
POST hook 2!	20
20

Which is most excellent.  Now your hook doesn't care if more things are added, it will remain to do the same thing regardless if Klei adds in more parameters to the function call.  This will NOT fix things if Klei removes arguments.  At that point you will need to do work to fix up the mod, no way around that as your base assumption of the function is no longer valid (the function having N arguments that are of a certain type and are doing a specific thing like adding them together).

 

However there is still an issue: Return values.  You're only returning one value, and the function now returns two values!  Gee Bill why can't you return two values?

Well you can, it's just that we're not handling it.  Like the arguments case, there can be an unlimited number of return values to account for.  You don't know how many Klei will add in the future though so to type them all out is a fool's errand.

In this case we'll encapsulate the return values into a new table and then unpack the table out in the return.

There's a function in LUA that is called 'unpack' (table.unpack in later revisions of LUA) which takes a table input and returns back all of the ipairs-iterable results as separate return values.

Example:

unpack({9, 5, 3})

Returns:

9, 5, 3

To be an ipairs-iterable table all of the keys of a table must be integers, sequential, and starts with the value of 1.

In this example table the table is equal to:

{
    [1] = 9,
    [2] = 5,
    [3] = 3,
}

So the table entries start a 1, add by 1 up to 3, and are all integers.  Thus this table is ipairs-iterable.

 

Notice how I made that table there by defining it as a table {#, #, #}.  Do you know what also uses commas for their values?  A return result.  "return #, #, #".

So you can encapsulate a function call with '{' and '}' to take all of the return values and put them auto-magically into a new table to use.

To return the values back out of the table you'll use this unpack function.

 

Let's put them into the function hooker:

local SomeFunction_old = SomeObject.SomeFunction
SomeObject.SomeFunction = function(self, arg1, arg2, ...)
    print("PRE hook 1!", arg1)
    arg1 = arg1 * 10
    print("PRE hook 2!", arg1)
    local ret = {SomeFunction_old(self, arg1, arg2, ...)}
    print("POST hook 1!", ret)
    ret = ret + 5
    print("POST hook 2!", ret)
    return unpack(ret)
end

Prints:

PRE hook 1!	1
PRE hook 2!	10
SomeFunction	10	2	3	4
POST hook 1!	table: 0x9f8fb0
input:15: attempt to perform arithmetic on a table value (local 'ret')

Egads!  We've just broken our assumption that the return result from the function call is an integer.  Instead 'ret' is now a table of return results, so we'll need to modify the operations after it to reflect that change.  We'll be modifying the first return result, so our index of 'ret' will be '1' to use.

 

So to fix it:

local SomeFunction_old = SomeObject.SomeFunction
SomeObject.SomeFunction = function(self, arg1, arg2, ...)
    print("PRE hook 1!", arg1)
    arg1 = arg1 * 10
    print("PRE hook 2!", arg1)
    local ret = {SomeFunction_old(self, arg1, arg2, ...)}
    print("POST hook 1!", ret[1])
    ret[1] = ret[1] + 5
    print("POST hook 2!", ret[1])
    return unpack(ret)
end

Prints:

PRE hook 1!	1
PRE hook 2!	10
SomeFunction	10	2	3	4
POST hook 1!	15
POST hook 2!	20
20	4

Marvelous.  Now our hooks will work if Klei adds more arguments or return results without having to update the mod to keep it up to date.

 

 

So, ultimately, to help future proof your mod you will hook functions by using this as a boilerplate:

local SomeFunction_old = SomeObject.SomeFunction
SomeObject.SomeFunction = function(self, arg1, arg2, ...)
    arg1 = 2
    arg2 = 4
    local ret = {SomeFunction_old(self, arg1, arg2, ...)}
    ret[1] = 0
    return unpack(ret)
end

Where arg1 and arg2 are your expected arguments that you know 100% of everything about.  If Klei edits these arguments then your assumption is broken and your mod might need updating to fix.  However, if Klei adds on additional arguments and leaves arg1 and arg2 alone then you need to do nothing to update your mod.

Likewise with return values, the ones you touch from the table 'ret' will have the assumption that the return result ret[1] is 100% known.  If Klei changes the return results around or their intrinsic value/type, then your assumption is broken and your mod may need an update to fix it.  However, if Klei maintains the first return result and adds more then your mod doesn't need an update.

 

 

Now, if you're not altering return values then you can simplify the ending bit by not packing and then unpacking the return results.

local SomeFunction_old = SomeObject.SomeFunction
SomeObject.SomeFunction = function(self, arg1, arg2, ...)
    arg1 = 2
    arg2 = 4
    return SomeFunction_old(self, arg1, arg2, ...)
end

 

Edited November 16, 2021 by CarlZalph
Evil tabs. No alter return values case.

[zetake]

Your tutorial is missing a thing. Handling empty return results.

I mean not every function has a return or there might be no need to return.

So I think using this would be more convenient.

if ret ~= nil then if type(ret) == "table" then if unpack(ret) ~= nil then
	return unpack(ret) end
else
	return ret end
end

 

Edited July 14, 2021 by zetake

[CarlZalph]

@zetake You don't need such checks, and it's safe to return nil.  This hooking method works with functions that don't return anything at all.

As for the 'no need to return nil' that's a false assumption.  The idea is to futureproof your function in case in the future it does change to return something.

A function that doesn't use the return statement still returns nil.

[Ultroman]

I love this write-up! In few words, you have connected so many dots that I'm even still not feeling 100% confident about, even after writing many mods. Thanks for this! It will surely help many people understand how to "think in Lua".

One thought: I can only hope that Klei adds new return values and parameters at the ends, keeping the "assumed structure", and I can only hope that those values still carry the same data, after a change has been made. If we're just slightly pessimistic, would it not be better to have the mod fail and tell you where it is failing, rather than having it working improperly in silence? I would actually rather be notified that some code that my mod is targeting has been changed, but alas, I have to check manually each update (using a diff tool on the previous game code). I get that that's a choice one has to make as a developer, whether to go for maximum uptime or maximum correctness, but I'm interested in your thoughts on this (and anyone else's, for that matter).

Happy New Years! 

Edited January 4, 2022 by Ultroman

[Wonderlarr]

On 1/4/2022 at 9:38 AM, Ultroman said:

I love this write-up! In few words, you have connected so many dots that I'm even still not feeling 100% confident about, even after writing many mods. Thanks for this! It will surely help many people understand how to "think in Lua".

One thought: I can only hope that Klei adds new return values and parameters at the ends, keeping the "assumed structure", and I can only hope that those values still carry the same data, after a change has been made. If we're just slightly pessimistic, would it not be better to have the mod fail and tell you where it is failing, rather than having it working improperly in silence? I would actually rather be notified that some code that my mod is targeting has been changed, but alas, I have to check manually each update (using a diff tool on the previous game code). I get that that's a choice one has to make as a developer, whether to go for maximum uptime or maximum correctness, but I'm interested in your thoughts on this (and anyone else's, for that matter).

Happy New Years! 

I'd like to know your process for using a diff tool, or well what exactly that is? I've never heard of it, does it check differences in code from one version to another?
 

Also, unrelated to the above, this is probably the most eye opening post I've read on the forums so far, I never did take the LUA crash course as I had a lot of prior knowledge and figured a lot out with intuition, this really changes how I see LUA and opens up SO MANY THINGS I CAN DO! Thanks!

[CarlZalph]

On 1/4/2022 at 12:38 PM, Ultroman said:

If we're just slightly pessimistic, would it not be better to have the mod fail and tell you where it is failing, rather than having it working improperly in silence? I would actually rather be notified that some code that my mod is targeting has been changed, but alas, I have to check manually each update (using a diff tool on the previous game code).

Ah well the outcome of using a function hook like this would be no different if there were changes made to the function's parameters the mod touches.

To prevent the mod's assumptions of the arguments or return values it directly touches from breaking things elsewhere, there's a thing called an assert.  It asserts a statement is true, and from there the rest of the code is safe to use that as an assumption basis.  If it fails, then it will generate a runtime error for inspection later.

Quote

assert (v [, message])

Issues an error when the value of its argument v is false (i.e., nil or false); otherwise, returns all its arguments. message is an error message; when absent, it defaults to "assertion failed!"

https://www.lua.org/manual/5.1/manual.html#pdf-assert

But using assert will incur performance costs as it's another function call.  However, it may not be needed for things which have a high probability of not changing.  Take an extreme example of some vector3 functions that do specific math related goodies.  They're very well defined and there's very little reason for them to change.  It would incur a lot of performance cost to check that the input arguments are what they should be if they were hooked as they're called all over the place.  User discretion is advised.

You may also chain your tests together in one assert for the arguments and one for the return values:

assert(type(arg1) == "number" and type(arg2) == "number")
assert(ret[1] == nil or type(ret[1]) == "number")

The first line demands arg1 and arg2 exist and are numbers, and the second line optionally requires ret[1] to exist and if it does then it must also be a number.

[Ultroman]

7 hours ago, TheSkylarr said:

Also, unrelated to the above, this is probably the most eye opening post I've read on the forums so far, I never did take the LUA crash course as I had a lot of prior knowledge and figured a lot out with intuition, this really changes how I see LUA and opens up SO MANY THINGS I CAN DO! Thanks!

You need to read the sacred Lua texts first, young padawan, or you might think Lua works like other languages, when it's a wild primordial beast in comparison :P

7 hours ago, TheSkylarr said:

I'd like to know your process for using a diff tool, or well what exactly that is? I've never heard of it, does it check differences in code from one version to another?

I bought Beyond Compare 4 and am loving it to bits. Yes, I keep the last extracted contents of the scripts.zip file in a folder, and then after an update, I extract the new scripts.zip file into a different folder, then right-click both and choose to use them for a comparison. It opens a side-by-side window of the folder structures, and it highlights any files with changes. It initially does just a checksum check, I think, but you can make it do a bitwise comparison to get more precision. Then I just have a separate list of the files I know I change things in or use critical things in for my mods, and check to see if any of those files have changed, and then I can double-click on the file to get a side-by-side view of the two files with the changes highlighted, with the option to only show the changed lines, to quickly get an overview. It can also be used to do 2-way and even 3-way merges, when used as an external diff-tool in for example SourceTree.

 

1 hour ago, CarlZalph said:

Ah well the outcome of using a function hook like this would be no different if there were changes made to the function's parameters the mod touches.

To prevent the mod's assumptions of the arguments or return values it directly touches from breaking things elsewhere, there's a thing called an assert.  It asserts a statement is true, and from there the rest of the code is safe to use that as an assumption basis.  If it fails, then it will generate a runtime error for inspection later.

But using assert will incur performance costs as it's another function call.  However, it may not be needed for things which have a high probability of not changing.  Take an extreme example of some vector3 functions that do specific math related goodies.  They're very well defined and there's very little reason for them to change.  It would incur a lot of performance cost to check that the input arguments are what they should be if they were hooked as they're called all over the place.  User discretion is advised.

You may also chain your tests together in one assert for the arguments and one for the return values:

assert(type(arg1) == "number" and type(arg2) == "number")
assert(ret[1] == nil or type(ret[1]) == "number")

The first line demands arg1 and arg2 exist and are numbers, and the second line optionally requires ret[1] to exist and if it does then it must also be a number.

Exactly, I wouldn't want to incur extra performance costs to do this, which is why I just accepted falling back on the diff-tool.

Edited January 11, 2022 by Ultroman

[tindomerel]

I'm attempting to follow this pattern using unpack as recommended. What's happening is this: attempt to call field 'unpack' (a nil value).

I understand that unpack has been moved from a global to table.unpack in newer versions of Lua (after 5.1). I tried table.unpack as well. Currently, I'm using this snippet I found recommended on StackOverflow for compatibility: 

table.unpack = table.unpack or unpack

It doesn't matter which way I try (unpack, table.unpack, or the above version meant to be compatible with both). I find unpack being used in the source code for DST, so I believe it should work. Here's a snippet showing where I'm trying to use it:

AddClassPostConstruct("components/builder_replica", function(self)
    local hasCharacterIngredient_old = self.HasCharacterIngredient

    self.HasCharacterIngredient = function(ingredient)
        local original_return = {hasCharacterIngredient_old(self, ingredient)}

        if ingredient.type == CHARACTER_INGREDIENT.WETNESS then
            print("check for wetness ingredient")
            if self.inst.components.moisture ~= nil then
                --round up wetness to match UI display
                print(self.inst.components.moisture.moisture)
                local current = math.ceil(self.inst.components.moisture.moisture)

                original_return[1] = current >= ingredient.amount 
                original_return[2] = current
            end
        end
        return table.unpack(original_return) --also have tried just unpack(original_return)
    end
end)

Does anyone know where I'm going wrong?

Edited October 7, 2022 by tindomerel
corrected one problem in the code (fixing it did not solve my problem though, so unrelated)

[Leonidas IV]

Hey @tindomerel, the unpack function isn't in your mod env, so you need to call GLOBAL.unpack instead of just unpack.