Migrate to LuaJIT?

[hyiltiz]

Been over 3 years since [1], so what are the roadblocks? Arguably LuaJIT got more mature but the game got richer so even more justifies aiming for performance.

 

 

[CarlZalph]

12 hours ago, hyiltiz said:

Been over 3 years since [1], so what are the roadblocks? Arguably LuaJIT got more mature but the game got richer so even more justifies aiming for performance.

From that thread:

Quote

I have actually had to modify lua scripts for LuaJit to behave 100% identical. Taking all this (and some other things) into account, the risk and added complexity has outweighed the benefit.

In short LUAJIT would require refactoring their code and could add in more bugs.

I don't blame Klei for not wanting to use it.

[hyiltiz]

That was LuaJIT 3 years ago. Assuming LuaJIT got more feature complete over these years, it may be possible to migrate with minimal effort? As I understand, it is quite common to hack lua the language, aka its interpreter, for lrge lua apps to expose or adjust some internal behavior. If Klei also uses those methods, I wonder if those are still possible to apply to LuaJIT.

[hyiltiz]

LuaJIT official site says [1] running lua 5.1 (DST runs lua5.1) compatible code with LuaJIT should require no changes to the source at all; that may not have been true a few years ago.

Quote

LuaJIT ought to run all Lua 5.1-compatible source code just fine. It's considered a serious bug if the VM crashes or produces unexpected results — please report this.

Known incompatibilities and issues in LuaJIT 2.0:

• There are some differences in implementation-defined behavior. These either have a good reason, are arbitrary design choices or are due to quirks in the VM. The latter cases may get fixed if a demonstrable need is shown.
• The Lua debug API is missing a couple of features (return hooks for non-Lua functions) and shows slightly different behavior in LuaJIT (no per-coroutine hooks, no tail call counting).
• Currently some out-of-memory errors from on-trace code are not handled correctly. The error may fall through an on-trace pcall or it may be passed on to the function set with lua_atpanic on x64. This issue will be fixed with the new garbage collector.

[1] http://luajit.org/status.html

Edited January 22, 2020 by hyiltiz

[hyiltiz]

I am asking [1] for a method to do this on client/user side but it seems (almost) impossible. If only Klei would compile DST against LuaJIT and see how it went.

[1] https://stackoverflow.com/questions/59893657/overload-inject-the-lua-runtime-of-a-compiled-program-with-custom-lua-luajit?noredirect=1#comment105915502_59893657

[PaintDream]

On 1/24/2020 at 11:12 PM, hyiltiz said:

I am asking [1] for a method to do this on client/user side but it seems (almost) impossible. If only Klei would compile DST against LuaJIT and see how it went.

[1] https://stackoverflow.com/questions/59893657/overload-inject-the-lua-runtime-of-a-compiled-program-with-custom-lua-luajit?noredirect=1#comment105915502_59893657

I've been trying to migrate LuaJIT for Dont'Starve via binary modding: https://github.com/paintdream/DontStarveLuaJIT  since 2016. It supports DS, DST and Dedicated Server for Linux now. 

This mod replaces builtin lua engine with LuaJIT 2. Various incompatibilities between LuaJIT and Lua are fixed. I would like to share my migration details/sugguestions if Klei were planning to do the same thing in official way.

(BTW, the most significant performance issue in Don't Starve may be the lack of GPU Instancing in rendering. It could be nice if Klei did it in official build.)

Edited May 8, 2020 by PaintDream
Fix spell.

[PaintDream]

On 1/22/2020 at 11:58 PM, hyiltiz said:

LuaJIT official site says [1] running lua 5.1 (DST runs lua5.1) compatible code with LuaJIT should require no changes to the source at all; that may not have been true a few years ago.

[1] http://luajit.org/status.html

That's not true. Don't Starve relies on serveral undefined behaviours of lua, which are implemented differently on LuaJIT.

For example, the strategy of constructing table with initializers is different, as well as string hashing algorithm. However some code requires a certain order when iterating a table. (e.g. RPC call id generation, mouse-clicking operation selection). As a result, the IDs of RPC are not consistent, leading connection failure as communicating with server / other clients.

In my implementation of DontStarveLuaJIT mod, I elimates these differences by porting lua's implementations to LuaJIT so it works fine. 

Edited May 9, 2020 by PaintDream

[hyiltiz]

That is great news! So glad to hear about your work. I checked out your Github project. Except what you have already mentioned (JIT performance and GPU instancing), would you like to elaborate what are the improvements over the official DS/DST build? It says it works for the Dedicated Servers only for Linux and the games for Windows; are you planning to support or maybe just write out a guide for macOS and Linux as well?

[PaintDream]

7 hours ago, hyiltiz said:

That is great news! So glad to hear about your work. I checked out your Github project. Except what you have already mentioned (JIT performance and GPU instancing), would you like to elaborate what are the improvements over the official DS/DST build? It says it works for the Dedicated Servers only for Linux and the games for Windows; are you planning to support or maybe just write out a guide for macOS and Linux as well?

I wrote a long summary on this mod but it's in Chinese: https://zhuanlan.zhihu.com/p/24570361 . Maybe you could read it with Google Translate (Highly recommended) ...

Here's the outline in English:

1. Locate lua functions by machine code searching (it's a really hard work...);

2. Solve the problem caused by missing of arg table in LuaJIT by inserting a statement to AST. (In lua, arg is usually a grammar sugar for { ... } which can be captured through closure, however LuaJIT does not support it).

3. Remove extra character-encoding support from LuaJIT. Lua suppresses slash '\' with illegal type. Therefore '\u' would not cause any error and would be converted into 'u' in Lua. However, LuaJIT support '\u1234' to indicate unicode characters. In some mods, we can find such a string 'use \umbrella to xxxx'. It is illegal in LuaJIT, causing exception and crashes the game.

4. Fix RunInSandboxSafe implementation.

5. Fix hash table iteration inconsistent problem by porting string hash algorithm and table insertion for initializer list. (Most interesting)

6. Fix different behavior for unpack() with 'nil hole' in it. e.g:

function foo(...)
	return {...}
end

print(unpack(foo(1, 2, 3, nil, 5)))

   Lua:

   1 2 3 nil 5

   LuaJIT

   1 2 3

7. Fix crash when deserializing a big save slot. LuaJIT only support at most 65535 constant in a lexical scope.

8. Fix Ice Flingomatic Trap failure by porting buggy version of table.reverse to luajit. (Yes, Ice Flingomatic Trap relies on table.reverse bug in Klei's implementaion)

Edited May 14, 2020 by PaintDream

[PaintDream]

7 hours ago, hyiltiz said:

That is great news! So glad to hear about your work. I checked out your Github project. Except what you have already mentioned (JIT performance and GPU instancing), would you like to elaborate what are the improvements over the official DS/DST build? It says it works for the Dedicated Servers only for Linux and the games for Windows; are you planning to support or maybe just write out a guide for macOS and Linux as well?

I am searching for a good injecting approach for Linux/macOS. As you know, the game requires steam to launch it (except Dedicated Server), so I need to inject my mod just before game starts. But by now I have no good idea. (Hacking game binary file would be OK but not portable for game upgrading and it's not a beautiful way)

[hyiltiz]

I skimmed through your original post in the Chinese site; it seems you had to rely on some reverse engineering to inject into the Windows binary which seemed to be a huge amount of work with too many detours and hassle. For Linux, I guess you can use NSA's Ghidra (though it's probably nothing platform specific)?

A more straightforward way would be to get some help from Klei devs, in the form of a few source code files (not even the whole source tree) that they deem relevant to patch against. Maybe even simpler: dynamic linking the lua library so you could simply update it with yours. You are essentially doing, to some extent, their job for them; they are busy providing content updates and couldn't focus much on optimization (they did a texture optimization recently so they do still care, it seems). Your project seems to have been working for about 2 years now, so if it passed some good testing (against the games and their popular and up-to-date mods), Klei devs may even be glad to incorporate your changes since your project is in MIT license.

Anyone knows which of the Klei devs we could AT here (@JoeW @bizziboi ?) to draw attention and seek for guidance and support?

Edited May 17, 2020 by hyiltiz

[PaintDream]

On 5/17/2020 at 8:30 AM, hyiltiz said:

I skimmed through your original post in the Chinese site; it seems you had to rely on some reverse engineering to inject into the Windows binary which seemed to be a huge amount of work with too many detours and hassle. For Linux, I guess you can use NSA's Ghidra (though it's probably nothing platform specific)?

A more straightforward way would be to get some help from Klei devs, in the form of a few source code files (not even the whole source tree) that they deem relevant to patch against. Maybe even simpler: dynamic linking the lua library so you could simply update it with yours. You are essentially doing, to some extent, their job for them; they are busy providing content updates and couldn't focus much on optimization (they did a texture optimization recently so they do still care, it seems). Your project seems to have been working for about 2 years now, so if it passed some good testing (against the games and their popular and up-to-date mods), Klei devs may even be glad to incorporate your changes since your project is in MIT license.

Anyone knows which of the Klei devs we could AT here (@JoeW @bizziboi ?) to draw attention and seek for guidance and support?

I could simply set LD_PRELOAD environment to load my shared library for linux (much easier than on Windows):

LD_PRELOAD=libpreload.so dontstarve_steam

I've published this mod 4 years ago to a Chinese Game Club called 'Don't Starve Game Bar'. All reported bugs that can be fixed were fixed. It's seems to be stable now but I still need a lot of tests.

It's really simple for Klei to integrate my mod: just clone my modified luajit version from repository and replace it. 

BTW: The graphic module is need to be refractoried to improvement performance. I could provide any help if Klei got any problems on shader optimization & GPU instancing.

Edited May 18, 2020 by PaintDream

[PaintDream]

On 5/17/2020 at 8:30 AM, hyiltiz said:

I skimmed through your original post in the Chinese site; it seems you had to rely on some reverse engineering to inject into the Windows binary which seemed to be a huge amount of work with too many detours and hassle. For Linux, I guess you can use NSA's Ghidra (though it's probably nothing platform specific)?

A more straightforward way would be to get some help from Klei devs, in the form of a few source code files (not even the whole source tree) that they deem relevant to patch against. Maybe even simpler: dynamic linking the lua library so you could simply update it with yours. You are essentially doing, to some extent, their job for them; they are busy providing content updates and couldn't focus much on optimization (they did a texture optimization recently so they do still care, it seems). Your project seems to have been working for about 2 years now, so if it passed some good testing (against the games and their popular and up-to-date mods), Klei devs may even be glad to incorporate your changes since your project is in MIT license.

Anyone knows which of the Klei devs we could AT here (@JoeW @bizziboi ?) to draw attention and seek for guidance and support?

By now, the only bug existed is that some mods use compiled-binary code from luac to "hide" thier details (luadec: ?), which are not recognized on LuaJIT. 

Edited May 18, 2020 by PaintDream

[hyiltiz]

Just tried it out your latest release from Github and is working fine in 64-bit Debian testing (Linux 5.6.0) with all the mods I have ever been using (about 10+). Game feels more fluent as well, and also taking less RAM it seems. Marvelous! 

Edited May 18, 2020 by hyiltiz

[PaintDream]

8 hours ago, hyiltiz said:

Just tried it out your latest release from Github and is working fine in 64-bit Debian testing (Linux 5.6.0) with all the mods I have ever been using (about 10+). Game feels more fluent as well, and also taking less RAM it seems. Marvelous! 

[CarlZalph]

On 5/14/2020 at 1:10 PM, PaintDream said:

I am searching for a good injecting approach for Linux/macOS. As you know, the game requires steam to launch it (except Dedicated Server), so I need to inject my mod just before game starts. But by now I have no good idea. (Hacking game binary file would be OK but not portable for game upgrading and it's not a beautiful way)

For OS X I've used rd_route by Dmitry Rodionov successfully before with osxinj to handle the bootstrapping.  Latest OS X versions may require function interposing, not sure.

// Other headers.

#include "rd_route.h"

// Function hook declarations.

void install(void) __attribute__ ((constructor));

void install()
{
    // Detouring and other tampering done here via rd_route.
}

sudo ./osxinj dontstarve_steam ./hooks.dylib

When I did it back in October 2018 Steam didn't complain and the dylib was loaded.

 

Interposing:

#define DYLD_INTERPOSE(_replacment,_replacee) \
    __attribute__((used)) static struct{ const void* replacment; const void* replacee; } _interpose_##_replacee \
    __attribute__ ((section ("__DATA,__interpose"))) = { (const void*)(unsigned long)&_replacment, (const void*)(unsigned long)&_replacee };

 

[PaintDream]

5 hours ago, CarlZalph said:

For OS X I've used rd_route by Dmitry Rodionov successfully before with osxinj to handle the bootstrapping.  Latest OS X versions may require function interposing, not sure.

// Other headers.

#include "rd_route.h"

// Function hook declarations.

void install(void) __attribute__ ((constructor));

void install()
{
    // Detouring and other tampering done here via rd_route.
}

sudo ./osxinj dontstarve_steam ./hooks.dylib

When I did it back in October 2018 Steam didn't complain and the dylib was loaded.

 

Interposing:

#define DYLD_INTERPOSE(_replacment,_replacee) \
    __attribute__((used)) static struct{ const void* replacment; const void* replacee; } _interpose_##_replacee \
    __attribute__ ((section ("__DATA,__interpose"))) = { (const void*)(unsigned long)&_replacment, (const void*)(unsigned long)&_replacee };

 

Thanks a lot!

I think I could try it if I got a Mac ...

[hyiltiz]

I have a Macbook Pro running Debian and another running latest macOS. I'd be happy to run/compile/test stuff out.

[hyiltiz]

WouldDYLD_INSERT_LIBRARIES environment variable also work for OSX/macOS? That probably means we can use the same injection method as in Linux, and just need to update the machine codes/hooks?

[1] https://stackoverflow.com/questions/8514783/what-is-the-exact-equivalent-to-ld-preload-on-osx

[vbvv]

It seems that some mods are compatible with Luajit.