[General] - Still unable to generate a world without patching the .exe

[outseeker]

Bug Submission:

Category: General

Issue Title: Still unable to generate a world without patching the .exe

Issue Description: With the game files verified by Steam, I can't generate a world. It crashes every time like this:

ModIndex: Load sequence finished successfully.

Reset() returning

DoLuaFile Error: not enough memory

not enough memory

Error loading worldgen_main.lua

WorldSim::SimThread::Main() ERROR

LUA: RUN-TIME ERROR not enough memory

Steps to Reproduce: Run the game and try to generate a new world.

As I expect you will be able to do that, the only other step I have ever performed is to run the generic 4GB patch on the game, allowing it to access RAM in higher addresses, then allowed Steam to revert that .exe back to default.

I can generate the world if I'm using the patched .exe but I can not when using the default distribution .exe

I get the same result whether I'm running mods or not but I've disabled all mods and re-run it for the attached log.txt

For reference, I have an i7 4930k with 16GB RAM, running Win7 x64

Cheers

[imsomony]

Attachments don't seem to work properly from the bug submission form initially - could you post it again?

[bizziboi]

Hi @outseeker, 

 

Hmmm, that's really odd - obviously a lot of people can run the game with the 2gig limit, so something is causing your system to use more memory.

 

Few questions:

- Could you upload your DXDiag?

- Do you have any mods enabled?

- Are you running fullscreen or windowed? What resolution?

 

Thanks

 

[Wade]

Can you also try unsubscribing from all of your mods, and then deleting your mods folder to see if it makes a difference?

 

Some mods, if outdated, make cause problems for your game when it tries to load them.

[outseeker]

@Wade, I am subscribed to quite a lot of mods so unsubbing then resubbing to them all isn't really too ideal, but I disabled all mods then exited the game, renamed the mods folder and had Steam verify the game cache before then running the game, which recreated the default mods folder and could generate a world just fine.

 

When I then went into the Mods menu, let it do its verification and downloading of subbed mods and such, didn't select any mods to enable at all and just hit Back, I was again unable to generate a world.

 

Why is anything from the mods folder being loaded when none of it is enabled? I will try and see if I can find which precisely is the mod in question though.

 

@imsomony, I've attached one to this post, sorry I didn't notice its absence in the first place (I think this log.txt actually shows my going through the process of allowing the Mods menu to do its thing then trying to generate a world, given the attachment size I just noticed)

 

@bizziboi, 1920x1080 fullscreen

log.txt

[outseeker]

I had planned to delete one mod folder at a time to determine which specifically is causing the issue, but starting top to bottom, removing 1 mod folder caused the game to hard crash at generation, removing another mod folder caused a similar Windows handling the crash, and removing another folder resulted in a nice in-game black screen with the nice styled borders, simply saying "out of memory".

 

At this point I wasn't really satisfied my method would properly indicate which mod is at fault, as I might just be lowering the memory usage to tolerable levels eventually, even with the defective mod still running. Deleting the bulk of the mod folders at once allowed generation to start working again, but I have perhaps 50 folders in there; any time efficient ideas? XD

[rezecib]

any time efficient ideas? XD

Binary search. I know Pond Manager's author was saying he was having memory issues with his mod, though. 

[outseeker]

OK, I've unsubbed from about 12 mods, each time re-opening the Mod menu in game to verify and download, then closing and restarting the game to try and generate a world. I found the game stopped saying out of memory when it crashed and started just hard crashing to a Windows close dead program box instead.

 

Finally I've unsubbed from the Regeneration mod and I can again generate a world.

When I resub to it, open Mods to let it download, close the game, reopen the game and try to generate a world, it hard crashes after character selection with:

 

[Scatter] SPAWNING PLAYER AT: (-370.19, 0.00, -149.95) Serialize session to session/03B000E49980C6EB/KU_JKCekuZKDeserialize local session session/03B000E49980C6EB/KU_JKCekuZKFailed to load minimap from session data: session/03B000E49980C6EB/KU_JKCekuZK_minimapAssert failure 'tex != NULL' at ..\source\renderlib\OpenGL\HWRenderer.cpp(855): Trace follows...

No trace follows unfortunately.

 

So.. Not exactly certain if it's the Regeneration mod doing something dodgy or if I just finally removed enough mods for the out of memory error to not occur at that particular point. Still wondering how disabled mods can break my game? ^_^

[outseeker]

@rezecib, not sure where to begin on the suggestion of binary search, but I unsubbed from Pond Manager and that does allow me to generate and enter a world so cheers for that mate. Not sure if that's because of the mod, or just because I lessened the load again tho.

 

*edit* @Wade, would love some info regarding how a disabled mod can break world gen! <3

[outseeker]

For reference we have this thread:
http://forums.kleientertainment.com/topic/47877-server-crash-world-generation-not-enough-memory/?hl=%2Bgenerating+%2Bworld+%2Bout+%2Bmemory

 

Why would a mod be unloaded and reloaded several times during world creation, causing this out of memory issue? Can something be done? <3

[Wade]

I'll talk with our programmer in charge of the mods system to see why mods data is being loaded upon game start-up.

[MentlegenScientist]

I'll talk with our programmer in charge of the mods system to see why mods data is being loaded upon game start-up.

I hope this will be fixed soon, because I'm having the exact same issue. Except, even without my mods folder, it crashes with this in the log.txt at the end.

 

Reset() returning

DoLuaFile Error: not enough memory

not enough memory

Error loading worldgen_main.lua

WorldSim::SimThread::Main() ERROR

LUA: RUN-TIME ERROR not enough memory

 

And that's with my mods all disabled and in a folder I've labelled mods_bak, so I have no idea what's going on...

[rezecib]

@MentlegenScientist, Try deleting your mod config data? Documents/Klei/DoNotStarveTogether/save/mod_config_data

[Kzisor]

I'll talk with our programmer in charge of the mods system to see why mods data is being loaded upon game start-up.

 

I can confirm that too many configuration_options can indeed cause the out of memory error. I've talked with PeterA about it and he said until there was a fix that we need to lessen the amount of configuration_options available.

 

Unfortunately for some mods that is an unreasonable request as certain mods, e.g. Pond Manager, is specifically designed to allow server hosts to configure their server to make ponds more difficult or easier based on the season. While I could in theory simply disable Spring and Autumn for DST, this is a global issue which happens in DS as well. Since my mod works for both games; I've decided to remove it from Klei and Steam until the issue has been resolved.

[outseeker]

I can confirm that too many configuration_options can indeed cause the out of memory error. I've talked with PeterA about it and he said until there was a fix that we need to lessen the amount of configuration_options available.

 

Unfortunately for some mods that is an unreasonable request as certain mods, e.g. Pond Manager, is specifically designed to allow server hosts to configure their server to make ponds more difficult or easier based on the season. While I could in theory simply disable Spring and Autumn for DST, this is a global issue which happens in DS as well. Since my mod works for both games; I've decided to remove it from Klei and Steam until the issue has been resolved.

 

*sadface*

 

Would you be in any way interested in releasing a mini-version that only enables you to break the ice in winter perhaps? I don't think there is another mod on the Workshop for this at all

[Kzisor]

@outseeker, I will look into it. I could easily pull that section of code out of the Pond Manager and make a small mod for that. 

[Wade]

Yup, we've talked about this issue in the office and we're looking into a fix for it.

[outseeker]

Thanks all! I would love a fix, or a mini Pond Manager- both/either would be awesome <3

[Maris]

I have the same issue. I'd like to join your "Out Of Memory" club. 

 

First of all, I found that too many objects with AddPrefabPostInit can causing this issue. For example, I tried to change loot from evergreens:

function evergr_init(inst)	local old_1 = inst.components.growable.stages[1].fn	inst.components.growable.stages[1].fn = function(inst)		old_1(inst)		inst.components.lootdropper:SetLoot({"log","log","log","log","log"})	end	local old_2 = inst.components.growable.stages[2].fn	inst.components.growable.stages[2].fn = function(inst)		old_2(inst)		inst.components.lootdropper:SetLoot({"log","log","log","log","log","log","log","log","log","log"})	end	local old_3 = inst.components.growable.stages[3].fn	inst.components.growable.stages[3].fn = function(inst)		old_3(inst)		inst.components.lootdropper:SetLoot({"log","log","log","log","log","log","log","log","log","log","log","log","log","log","log","pinecone"})	endendAddPrefabPostInit("evergreen",evergr_init)

​

This code causing crash without any other mods. However it was easy to fix it:

local EVER_LOOT={	{"log","log","log","log","log"},	{"log","log","log","log","log","log","log","log","log","log"},	{"log","log","log","log","log","log","log","log","log","log","log","log","log","log","log","pinecone"},}function evergr_init(inst)	local old_1 = inst.components.growable.stages[1].fn	inst.components.growable.stages[1].fn = function(inst)		old_1(inst)		inst.components.lootdropper:SetLoot(EVER_LOOT[1])	end	local old_2 = inst.components.growable.stages[2].fn	inst.components.growable.stages[2].fn = function(inst)		old_2(inst)		inst.components.lootdropper:SetLoot(EVER_LOOT[2])	end	local old_3 = inst.components.growable.stages[3].fn	inst.components.growable.stages[3].fn = function(inst)		old_3(inst)		inst.components.lootdropper:SetLoot(EVER_LOOT[3])	endendAddPrefabPostInit("evergreen",evergr_init)

Well, this is better, because there is no crash. But there are still too many functions that are cached. This PostInit function executes for EVERY evergreen in the world. So it eats some memory.

 

But it can be optimized better. Let's hide this code in the function of component:

local EVER_LOOT={	{"log","log","log","log","log"},	{"log","log","log","log","log","log","log","log","log","log"},	{"log","log","log","log","log","log","log","log","log","log","log","log","log","log","log","pinecone"},}dropper = _G.require "components/lootdropper"gen = dropper.GenerateLootfunction dropper:GenerateLoot(...)	local loot = gen(self,...)	if loot[1] == "log" then		--count logs		local cnt = 0		local twigs = false		for i,v in ipairs(loot) do			if v=="log" then				cnt = cnt+1			elseif v=="twigs" then				twigs = true			end		end		--x5		if cnt>=1 and cnt<=3 then			loot = EVER_LOOT[cnt]		end		if twigs then			table.insert(loot,"twigs")			table.insert(loot,"twigs")		end	end	return lootend

Well, this is not good code hack. But this code is cached only ONCE. Even more: this code doesn't execute before game starts.

 

And now I'm stuck again. I need to count every prefab in the world (for optimization purposes). So I wrote code that counts prefabs:

local spawn_counter = {}local function AddCounter(prefab)	spawn_counter[prefab] = 0	AddPrefabPostInit(prefab,function(inst)		spawn_counter[prefab] = spawn_counter[prefab] + 1		inst:ListenForEvent("onremove", function()			spawn_counter[prefab] = spawn_counter[prefab] - 1		end)	end)endAddCounter("evergreen")AddCounter("grass")AddCounter("sapling")

​

And this code eats too much of memory, causing crash "out of memory". And I'm not sure how to optimize it, because there is no way to skip AddPrefabPostInit in this case.

 

One thing that I can't figure out: WHY is it caching? There is no DoPeriodicTask and so on in AddPrefabPostInit. So PosInit function should leave the cache after executing.

 

I hope there will be a patch of the game which could fix this issue.

[bizziboi]

Hey @Maris,

 

If you could send me a simple mod (or mods reproducing the behaviour you showed above) that causes this to happen I will have a deeper look at this.

 

 

[bizziboi]

Hey @Maris, 

 

Okay, yes there will be a patch to address memory issues most likely. I am still working on it.

 

Your last sample (the spawn_counter one) doesn't cause any memory issues for me whatsoever, so not sure what is up there.

 

The previous one, while the patch would address it crashing, it still takes the loading time from a few seconds on my machine to well over a minute since there's a bug in that code (cheers to @V2C for spotting that). Also, a fixed version doesn't run out of memory for me (but I wasn't running any other mods so your mileage may vary).

 

You give evergreens a closure calling their old growthstage function and assign that to their growthstage function. Since all evergreens share the same growthstages table, you're basically creating a longer and longer callchain on every prefab spawned. The second prefab's closure will call the first prefab's closure which will call the original. The third evergreen will create a closure that calls the second one that calls the first one that calls the original - you get the idea.

 

In order to make this work yet be compatible with other mods that might change individual growthstage functions you'd probably want to create a dictionary of what is already wrapped and what not, and if a tree is pointing to an unwrapped growth stage, wrap it in your function and set that wrapped function and store the wrapped off for future re-use.

Something like

local growthfuncs = {}

 

local function evergr_init(inst)

 

    local old_1 = inst.components.growable.stages[1].fn

    local old_2 = inst.components.growable.stages[2].fn

    local old_3 = inst.components.growable.stages[3].fn

 

    if not growthfuncs[old_1] then

         local func = function(inst)

             old_1(inst)

             inst.components.lootdropper:SetLoot({"log","log","log","log","pigman"})

        end

        growthfuncs[old_1] = func

        growthfuncs[func] = func -- basically say I am wrapped already, prevents need for another dictionary

    end

    if not growthfuncs[old_2] then

        local func = function(inst)

            old_2(inst)

           inst.components.lootdropper:SetLoot({"log","log","log","log","log","log","log","log","log","log","merm"})

        end

        growthfuncs[old_2] = func

        growthfuncs[func] = func

    end

    if not growthfuncs[old_3] then

        local func = function(inst)

            old_3(inst)

                 inst.components.lootdropper:SetLoot({"log","log","log","log","log","log","log","log","log","log","log","log","log","log","log","rabbit"})

        end

        growthfuncs[old_3] = func

        growthfuncs[func] = func

    end

 

    inst.components.growable.stages[1].fn = growthfuncs[old_1]

    inst.components.growable.stages[2].fn = growthfuncs[old_2]

    inst.components.growable.stages[3].fn = growthfuncs[old_3]

end

 

AddPrefabPostInit("evergreen",evergr_init)

[Maris]

@bizziboi, I solved my memory issues and ingame lags. I just unsubscribed from all mods, and deleted mods folder. The I started from zero, subscribing only useful mods. So I'm not sure which mod exactly have an issue. But it seems that my mods are ok now and the game ok too.

And thank you for noticing a bug in my code. I don't thought about that. I was really stupid if missed it.