more than 30GB on ONI Appdata output_log.txt

Hi, if you have a text editor can handle it, can you open that file and just capture the last 1000 lines or so of it? It is very likely that there's just a single log getting spammed in there.

I dont think a normal computer can open a >30GB file in a text editor within a reasonable time. It would have to be loaded into RAM (8 or 16GB are typical sizes) and then the system swaps to harddisk. It would take >hour...

For me, this sounds like a windows problem, maybe with permissions. It could be, the file is writeable, but it wont get deleted. So the game just adds data, and the file grows bigger and bigger. A few Megabytes are normal for heavy gamers, but not gigabytes.

yep is impossible to open this Log on any type of text editor, world, notepad, notepad++, none o these can, and 37gb of data in less than 30 minutes? i don't think is just repeating the same line every time... the problem is see if is a mod incompatibility problem or a base game problem, maybe is a base game problem that gonna this scale because of mods, i don't know, will need 64gb of ram to see this text... and i only have 16gb

Edited February 7, 2020 by Umbra_Makiavel

Well, some editors definitely can 

However, if you try compressing the file, that is another option for uploading the log.

so what type of editor i should use? and how do i compress 37GB to 99MB?

Don't worry about the editor, it's probably not worth the time.

Any sort of compression will probably do the trick, I've seen a 80GB output_log.txt get compressed to 42MB.  If you'd like a recommendation for compressing, 7-Zip is free and open-source.

Thank you very much for taking the time! I know dealing with these large files is frustrating.

ok i compressed with Winrar and is now possible to send here. good luck.

output_log.rar

@fatheroctopus can you make the game rotate the log file or compress it automatically when it reaches a certain size? That would prevent such issues from happening  

I bet ya the log file contains thousands of lines like these:

[21:18:22.890] [13] [WARNING] Effective mass cannot be greater than capacity!

I may be wrong on this, but i never had a log file > serveral MB. And this in years. So i assume it will be deleted under normal circumstances.

I think you need a log rotation. It is no matter, which message spams into log. But i brought here end of log file. Also, Lister in Total Commander can read big files. Do not use editors, use readers. And there are log: 

[20:08:05.180] [14] [ERROR] [PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
[PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 45)

[20:08:05.180] [13] [ERROR] [PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
[PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 45)

[20:08:05.180] [15] [ERROR] [PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
[PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 45)

[20:08:05.180] [14] [ERROR] [PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
[20:08:05.180] [13] [ERROR] [PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
[PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 45)

[PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 45)

[20:08:05.180] [15] [ERROR] [PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
[PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 45)

[20:08:05.181] [16] [ERROR] [PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
[PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 45)

Game.OnApplicationQuit()
Setting up 1 worker threads for Enlighten.
  Thread -> id: 6b0 -> priority: 1 
[20:08:05.680] [1] [INFO] [PLib/EfficientFetch] Destroying EfficientFetch

 

This is modding related iirc, interesting to note.

PLib

now the problem is, mods caused this or mods showed what can't be seen before?

I think it's safe to say that the mod PLib/EfficientFetch is causing the issue in this case.

Changed Status to Closed

For reference, the beginning of the spam looks like this:

[06:19:25.514] [1] [INFO] [PLib/SweepByType] Replacing sweep tool 9 with filtered sweep
[48;5;24m  [ResearchManager]: Deactivating research watch [48;5;0m
[48;5;24m  [ResearchManager]: Deactivating research watch [48;5;0m
[06:19:27.603] [1] [INFO] Screen resolution updated, saving values to prefs: 1920x1080 @ 60, fullscreen: True

Unloading 6 unused Assets to reduce memory usage. Loaded Objects now: 216004.
Total: 686.759108 ms (FindLiveObjects: 13.372506 ms CreateObjectMapping: 9.368581 ms MarkObjects: 663.507024 ms  DeleteObjects: 0.510034 ms)

[06:19:29.175] [13] [ERROR] [PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
[06:19:29.175] [14] [ERROR] [PLib/EfficientFetch] System.MissingFieldException Field 'FetchManager/Fetchable.masterPriority' not found.   at PeterHan.EfficientFetch.EfficientFetchManager.UpdatePickups (FetchManager+FetchablesByPrefabId fetch, Navigator navigator, UnityEngine.GameObject fetcher, System.Collections.Generic.IDictionary`2[TKey,TValue] cellCosts) [0x0003d] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 
  at PeterHan.EfficientFetch.EfficientFetchPatches+FetchablesByPrefabId_UpdatePickups_Patch.Prefix (FetchManager+FetchablesByPrefabId __instance, Navigator worker_navigator, UnityEngine.GameObject worker_go, System.Collections.Generic.Dictionary`2[TKey,TValue] ___cellCosts) [0x00018] in <cb7314d5b0da45e98c8f6dad31c080f8>:0 

 

so i still have a problem... i don't have any mod called Efficient Fetch or similar thing...

I have 2 gigs of log file within 2.5 minutes (95% of that is just loading a save) so it definitely would benefit from being gzipped and rolled over. I know that's a builtin feature of slf4j api in Java, I assume there's something similar in .NET.

Mine's 59 thousand lines of "[06:43:04.008] [1] [INFO] SolidVentComplete" etc for probably every tile and entity in the game. I'll have to figure out where it's coming from. Whether the offenders are mods or not, no log file needs to take that much space, especially when streaming gzip exists.

I'd start directing this file to a RamDisk so it doesn't obliterate my SSD, if I didn't think it'd quickly overrun the 10-15 gigs of ram I have available while ONI is loaded.

Edited February 15, 2020 by FinalDoom