Why an optimization update is likely

[calibayzone]

The devs really don't want to affect balance that much for future updates. People will complain that "x design doesn't work anymore/isn't worth it", or "x design is now TOO good, feels exploity". So they focused on features, bugs, and balance for the last few months leading up to the official release and mostly neglected optimization.

However, no one will complain if the game is made faster. On top of this, we are complaining quite loudly about how slow things are, and if factorio is a (very) rough reference point of how fast things could be, even 10% of factorio would be 60fps on 3x speed with no critter-ranching lag.

[Gurgel]

I don't think you can compare ONI and Factorio in that regard. Factorio has almost no planning component. The only thing I see is critters and they behave in an extremely simple way. Factorio also has basically no Physics. ONI, on the other hand, does excessive planning all the time, because it would not work otherwise and has a lot of physics. 

[natanstarke]

The comparison with factorio is invalid indeed. Idk if its even possible to optimize oni more but if it is i hope its made true.

[bobucles]

Easy. Efficient. Flexible. You only get to pick one for your code.

Players don't have access to the debug tools, so we'll never know where the CPU cycles end up. But there are two known systems that are understandably high demand. The main world simulation has to process the entire world every tick, and the agents in the world also have a lot of data crunching that goes into their decision making. It's difficult to analyze anything beyond that without the source code or any CPU trackers or any real information at all.

Until ONI has a fairly decent CPU tracker like this, everything else is blind shooting in the dark speculation and hearsay.

[nakomaru]

Well, yes. Release means no major content patches until DLC. Only bug fixes, small balance changes and optimization. How much we get of those depends on how many resources they have available, though. I don't think we are ever going to get the dream of fully parallelized GPU-offloaded compute.

[Pentalis]

2 hours ago, nakomaru said:

Well, yes. Release means no major content patches until DLC. Only bug fixes, small balance changes and optimization. How much we get of those depends on how many resources they have available, though. I don't think we are ever going to get the dream of fully parallelized GPU-offloaded compute.

But we will get the dream of stuttering-causing glitches and bug-induced lag fixed, and that's already a lot!

[miauly]

2 hours ago, bobucles said:

Until ONI has a fairly decent CPU tracker like this

The image is mocked I assume or do you know a way to profile like that? 

I agree that it's not possible to speculate about performance degradation causes - either profiling or very specific sequence of steps to reproduce are needed. With the latter we can try to help, the thread with saves on rockets and digging a bottom of water reservoir is a great example. It's hard to isolate causes enough like that without profiling though.

But profiling is not an easy cure-all either. Unfortunately in a lot of cases you see too many stuff there with nothing standing out, then profiling can be inaccurate when we talk about lots of things happening really quick, then instrumentation required for profiling can alter the performance a lot..

I think as a community we can help if we work on isolating causes. Not just telling "if you tile the whole map it helps", or "if you get debris in certain order it helps" - a save should be prepared with certain FPS in "bad" state and certain higher FPS in "good" state, and the smaller the change between the two, the easier it would be for developers to use this to optimize the game. Ideally, it would be good if we could crowd-source awful saves that are really slow, share them here and try ways to improve them, and then file as bugs when we isolated the cause (causes). For example, a bug in pokeshell aggression was fixed like this during preview, exactly thanks to this sequence. Awfully slow save was posted as a bug - players crowdsourced that critters removal helped a lot - developers quickly isolated what was slowing things down and fixed that.

[calibayzone]

"has a lot of physics"

It has pseudo-physics, a discrete simplistic model. It doesn't model Navier Stokes or rigid bodies (Bullet physics). This is far easier and much more factorio-like, namely the amount of computation per grid cell isn't all that great. The heat transfer calculation is a simple formula applied each time step, so it may double the cost in comparison to active factorio areas but isn't overwhelmingly expensive.

"The main world simulation has to process the entire world every tick"

A 256 by 384 world, and keep in mind much like factorio many aspects of the world don't change every tick (gas exchange takes 1 second on average). In a Factorio megabase, even a "smallish" one, the region of "stuff actively happening" is far larger, even if it is more spread out into an infinite world. To get a sense of scale, and note that it is smooth while being recorded:
https://www.youtube.com/watch?v=jjtXHsv5E6M

"Factorio has almost no planning component."

Factorio robots have simple pathfinding (I assume planning=pathfinding), although you have thousands in megabases. Trains, on the other hand, have much more complex pathfinding. Only duplicates (of which we rarely have more than 50 or so) need complex path finding, simpler than factorio Trains in some sense because they don't collide with each other. Critters can be dumb and aimlessly drift around most of the time. Only if within 8 blocks of food or when called by a rancher do they need to do much, and even then they move very short distances. You can cache which blocks are "close to food", etc.

"Easy. Efficient. Flexible. You only get to pick one for your code."

This. Now is the time to focus on "efficient" as we don't need "flexible" as much now that the game mechanics settled down.

[Yoake]

Anyone that says that factorio doesn't require planning hasn't played it very long. They're very different games but Oni's world is very small at the end of the day.

Edit: What I mean is that sure, per tile Oni has a lot more calculations but with how massive factorio worlds can get, I don't buy that as a good excuse, nor should people make excuses for the devs to not constantly be optimizing or downplaying other games to make them comparable.

[Junksteel]

3 minutes ago, Yoake said:

Anyone that says that factorio doesn't require planning hasn't played it very long. They're very different games but Oni's world is very small at the end of the day.

Factorio looks like SNES Sim City at the end of the day.

These comparisons are pure speculation without the source code, right?

[Yoake]

2 minutes ago, Junksteel said:

Factorio looks like SNES Sim City at the end of the day.

These comparisons are pure speculation without the source code, right?

Who cares what the games look like? I've never seen anyone talk about graphics being a source of lag. Yes, it is speculation without the source code but number of calculations can be speculated.

[Junksteel]

Hey, don't get me wrong. I want the optimizations too. I'm just pointing that this comparison is being brought fairly often and it doesn't seem to add much to the discussion except illustrating how ONI is supposedly poor coded. It doesn't seem a fair comparison just as the graphic one is totally treacherous.

[Yoake]

Okay that's fair. I don't in any means want to claim that ONI is poorly coded, just want to push for even more improvements 

[Nebbie]

I think an optimization patch or five are likely, but I'd expect the next update will focus more on solving the current bizarreness of cooling solutions (there are still some really ridiculous ways to cool your base, including crazy exploits, and yet also the intended cooling solutions are a very narrow path of steam turbine setups with little to no deviation).

[cpy]

In normal game of factorio you have 1000s of moving parts thin of oni having 1000s of sweeper arms. Also you'd have so many transport belts it would make you sick. Factorio is about insanely large scale operation on insanely big map but map is not simulated for pressure or temperature so you don't have that. But you still have 1000s of moving parts.

[bobucles]

17 hours ago, miauly said:

The image is mocked I assume or do you know a way to profile like that? 

That's the Factorio profiler, there's nothing fake about it. Look at how small those numbers are, my pitiful 80k smelting/second base barely puts a dent in it.

Profilers are made the exact same way as any other tool, by programming them. Getting the precision down to fractions of a millisecond is its own trick though and not every library will get you there.

17 hours ago, miauly said:

then instrumentation required for profiling can alter the performance a lot..

No it's not terribly obstructive to the code. It reads two numbers, subtracts them, and the result is just a number. Computers handle billions of numbers every second, if a single subtraction pushes it over the edge then it never stood a chance.

[calibayzone]

"I don't in any means want to claim that ONI is poorly coded, just want to push for even more improvements"

It's more accurate to say "Factorio is lightning-fast" than "ONI is slow". Other games like dwarf fortress and minecraft have CPU fps issues as well at moderate scales, but the amount of optimization that Factorio underwent is second to none so fps issues only occur at insane scales. We don't need nearly that amount of optimization for the default world size, rather take it as inspiration of what can be done.

[miauly]

1 hour ago, bobucles said:

No it's not terribly obstructive to the code. It reads two numbers, subtracts them, and the result is just a number. Computers handle billions of numbers every second, if a single subtraction pushes it over the edge then it never stood a chance.

And this number is either logged or stored for your future analysis, which is an IO operation likely to go outside of CPU cache or push out actual computation data out of said cache. Now if you want to do a lot of them for a whole lot of things you do on said CPU, your program will slow to a crawl. Which is why there's a technique of stack sampling instead, that's used to look when you don't know where exactly to look. 

Also note that Factorio uses its own game engine, which means that developers can instrument all of the code that runs directly, by simply adding subtraction statements as you suggest. ONI uses Unity, so it's likely that the engine runs the code written by developers that's hooked into it, and to get a whole picture they need to instrument both the engine and their code together. Unity comes with profilers to address that, but these tools are likely using generic instrumentation techniques, as they need to work with any game code completely unknown to profiler. Which, inferring from my experience of using another generic profiler tool for another domain, may cause things to run so slow that a cycle of ONI running under profiler can slow to a point of simply freezing or crashing due to too much data being collected and saved for analysis.

[Gurgel]

13 hours ago, cpy said:

In normal game of factorio you have 1000s of moving parts thin of oni having 1000s of sweeper arms. Also you'd have so many transport belts it would make you sick. Factorio is about insanely large scale operation on insanely big map but map is not simulated for pressure or temperature so you don't have that. But you still have 1000s of moving parts.

You have. And all are nice little deterministic code snippets that you can just work through in a long list and parallelize with almost no loss of computing power. Path-finding on ONI is fundamentally different. Physics (as contained in ONI) is somewhat similar. 

My point is that the Factorio basically has no heavy lifting computation-wise. It does, of course, still consume lots of CPU, because it can have a massive number of small tasks, far more than ONI. But optimizing these things and parallelizing them is pretty easy (in comparison). ONI has some things that are complex and need complex code. Real-time path-finding and task scheduling are the primary examples. And these are really hard to optimize, in particular because you have to do things like trade speed for result quality, have a priority system (dupes dropping stuff), think about locality, etc. It does get very complex and whatever you do, some people will complain because you really cannot solve things optimally and precisely anymore.

Not saying that the current state of Factorio is not very impressive, because it is. But it is very different from ONI on the computing side.

4 hours ago, calibayzone said:

"I don't in any means want to claim that ONI is poorly coded, just want to push for even more improvements"

It's more accurate to say "Factorio is lightning-fast" than "ONI is slow". Other games like dwarf fortress and minecraft have CPU fps issues as well at moderate scales, but the amount of optimization that Factorio underwent is second to none so fps issues only occur at insane scales. We don't need nearly that amount of optimization for the default world size, rather take it as inspiration of what can be done.

I completely agree to that. I do believe I could write something like the Factorio Engine with comparable performance, but I do have more than a little experience with high-performance code and I would probably take half a decade to get there or longer. In particular because it would really be development from scratch, completely optimized down to things like cache-awareness and completely optimized for this single application.

That said, the performance ONI gets is also pretty impressive. But especially some Intel setups are pretty bad at smooth multitasking and due to the high interactivity and lots and lots of really excellent animations and some complex scheduling and planning tasks, this is very noticeable. 

6 hours ago, bobucles said:

Profilers are made the exact same way as any other tool, by programming them. Getting the precision down to fractions of a millisecond is its own trick though and not every library will get you there.

Hahaha, tell me about it. The last time I did try to optimize things with a profiler, quite a bit of my code did not even show up because it was too fast. (Well, the snippets, not the overall thing.) I also had the problem that the profiling calls themselves had a real impact, both from their computations and because they did increase the cache-footprint of the code. I had to go back to running synthetic benchmarks and trying things out. Of course, than only works (somewhat) well with very regular computations.

[cpy]

Path finding should be threaded too! I mean dupes are pretty much self contained individuals, you can do their calculations in parallel probably. Think of having base with only 1 dupe performance wise as long as you have enough CPU cores. I say 8C16T CPU would allow you for some insane populations.

Thing is that factorio dev team have one complete and utter insane person addicted to optimizing code, and it really shows his passion to tweaking code to insane levels. I mean it's not that uncommon for programmers to pull magic levels of performance out of system.

It used to be the norm when first gaming consoles had less RAM than display needed to use as VRAM buffer, CPU cycles weren't enough to do even simplest game. But they used to do some insane math magic and memory trickery to fit game in and make it run. Now when you have computer that has computational power so high that you don't even bother optimizing much, just let it do the thing.

Almost every piece of modern code can be optimized like crazy (provided you start with fancy unoptimized code). Unless your CPU is at 100% then we can still make it faster, and if CPU is at 100% then there is always some neat trick that can make things run faster.

ONI use only 20% of my CPU on my latest 500 cycle 20 dupes save. I think we can shove more optimizations there right?

[bobucles]

On 8/8/2019 at 2:38 PM, Gurgel said:

My point is that the Factorio basically has no heavy lifting computation-wise.

I dunno about that. If their developer FFF writeups are any indication, there's an enormous amount of heavy lifting that goes into every inch of code. There's tons of blogs talking about building O(n) code, caching data for fast access, rewriting graphical engines, and finding clever tricks that allow tens of thousands of entities to operate with the fewest CPU cycles possible. They've reached the point where threading struggles to contribute anymore, because they're blazing through the data so quickly the game ends up RAM throttled.

 Optimizing shows as both their passion and obsession. It is a clear example that anyone who thinks they can't find more CPU cycles is not just wrong but WRONG by several orders of magnitude.

[Nightinggale]

2 hours ago, cpy said:

Unless your CPU is at 100% then we can still make it faster, and if CPU is at 100% then there is always some neat trick that can make things run faster.

That's not true. Depending on the type of task it might not be doable to spread it across more CPU cores. If you do, the cores will be fighting for control of the same part of the memory and then you can slow everything down. One reason why single core games are popular is because you don't have to deal with cores fighting over control of the specific parts of the memory.

Imagine you have a restaurant with unlimited number of tables. You have waiters going between the kitchen and the tables, but there is only one door to the kitchen. If you have one waiter, the door is always free. If you have two then there is a risk of waiting to go through the door. As you increase the number of waiters, you end up in a situation where adding more won't make you serve food faster because they are queuing up at the door. Now if you could install a door for each waiter, then serve speed would scale according to the number of waiters.

It's precisely the same with CPU cores. If you can assign the cores with multiple tasks where they do not use the same memory (shared memory is ok if it's read only for all cores) then multi core is the way to go. If they have to write to the same memory, then you need some sort of "traffic control", which makes the cores wait and it can make cores wait so much that a single core without any "traffic control" would be faster.

This means you can't assume a game like ONI to be faster by just using more cores. It needs a design structure, which is multi core friendly too. Creating such designs is a lot harder than designing something, which only works on one core and sometimes it's just not possible. I agree ONI should use all cores and ideally the GPU too if possible, but assuming it's possible to use all cores 100% is a false assumption and we likely will never end up there.

Also let's not forget that single core code can be optimized too. Sometimes it's better to stick to one core, but reduce the load on that core by say 30% than it is to spread the workload on 4 cores.

[Sasza22]

4 hours ago, cpy said:

Path finding should be threaded too!

Isn`t it already? I remember something in the patchnotes back in QoL2 or 3.

[cpy]

1 minute ago, Sasza22 said:

Isn`t it already? I remember something in the patchnotes back in QoL2 or 3.

One thing is run pahfinding in one separated thread another thing is run pathfinding multithreaded.

At this point im just friggin guessing what needs to be improved, we don't have any detailed peformance info so we can't check what the hell is causing that stupid endgame lag. Pathfinding? Physics? Pipes? I don't know!

[Gurgel]

4 hours ago, bobucles said:

I dunno about that. If their developer FFF writeups are any indication, there's an enormous amount of heavy lifting that goes into every inch of code. There's tons of blogs talking about building O(n) code, caching data for fast access, rewriting graphical engines, and finding clever tricks that allow tens of thousands of entities to operate with the fewest CPU cycles possible. They've reached the point where threading struggles to contribute anymore, because they're blazing through the data so quickly the game ends up RAM throttled.

You misunderstand. By "heavy lifting" I mean complex calculations that cannot really be broken down, for example planning and scheduling in not very regular environments. Factorio does a heavy amount of "light lifting". That is different in that it is actually massively more regular and massively easier to optimize and parallelize. Not saying that it is easy, it is more the difference between "pretty hard but possible" and "basically impossible". 

Think of it as moving 100 t of rock by hand. If it is one block, you need "heavy lifting" and it becomes impossible. If it is pebbles you need patience, dedication and a few weeks or months of time.