A plight for performance improvements

[EnTro]

Oxygen not included is an amazing game and the intricate simulation of many interconnected systems is really the strength of this game. Performance slowly and relentlessly tanks as the game goes on and I see people with really high end PCs reporting the same on the steam forums. I'm really hoping that the developers will take some time to work on improving performance. I will not pretend like I know enough about the architecture of ONI to make suggestions, but I assume there must be something that can be done. I love the late midgame and basically always run 3x speed to keep the game progressing with big builds, but 15 FPS is really at the lower limit of what I find enjoyable (isn't simulation and render speed separated?).

What performance issues have you run into?

Are the developers willing to make some kind of official statement?

Spoiler

As people will like to know: i7-8700K, 32 GB Ram with RX 7700XT

 

 

[SharraShimada]

Oni is suffering from performance issues due to old decisions for stability.

You may try

 

Regarding your computers specs. Your GPU is 100% irrelevant. You could run ONI with your CPUs iGPU without any noticable difference.

Your 6 Cores also do not matter. Oni ist multhithreaded yes, but it needs ONE very potent core for the core tasks of the game. Throwing more cores at it, wont work.

 

And when it comes to general performance of the game: There are do´s and donts. NO multiple paths to any given location. It just results in pathfinding overloard. No debries on floors. Sort your containers. And solid tiles are better than excavated ones, unless you do a vaccuum there, for heat calculation reasons.

Combined with the mod i posted above, its pretty easy to run the game for several thousand cycles.

[Gurgel]

I do not have any performance issues in ONI since I upgraded to an AMD 7800X3D. I usually play classic-start, so large bases.

Also note that Intel gives a much worse experience at lower FPS than AMD does, due to inferior task-switching, so 30 FPS on AMD may feel more like 60FPS on Intel. 

[EnTro]

I appreciate the feedback. It is great that a performance mod exists, and it is helpful. It shows that the game can absolutely be optimized quite a bit, even with the current game engine as is. And this is only with a mod. With full access to the core code of the game quite a bit more can likely be done.

While I understand that a faster (single core performance) CPU would help, it's not the right solution. It's like putting a bigger engine in your car because the parking break doesn't turn off and slows down your car.

ONI was released in 2017. PCs shouldn't struggle to run a mid size base 7 years on. On the Steam forum a user with a monster PC with 14900K CPU reportedly still gets bogged down quite a lot. The solution is in code optimization.

[SGT_Imalas]

14 hours ago, EnTro said:

With full access to the core code of the game quite a bit more can likely be done.

all mods have that access

[SharraShimada]

23 hours ago, EnTro said:

I appreciate the feedback. It is great that a performance mod exists, and it is helpful. It shows that the game can absolutely be optimized quite a bit, even with the current game engine as is. And this is only with a mod. With full access to the core code of the game quite a bit more can likely be done.

While I understand that a faster (single core performance) CPU would help, it's not the right solution. It's like putting a bigger engine in your car because the parking break doesn't turn off and slows down your car.

ONI was released in 2017. PCs shouldn't struggle to run a mid size base 7 years on. On the Steam forum a user with a monster PC with 14900K CPU reportedly still gets bogged down quite a lot. The solution is in code optimization.

You cant optimize code beyond a certain point whithout compromising stability and compatibility. The Devs need the game to run on every PC. Peter (the mod author) on the other hand can just ignore some problems and be fine with it. He does not need to satisfy paying customers.

 

[pether]

1 hour ago, SharraShimada said:

Pether (the mod author) on the other hand can just ignore some problems and be fine with it.

Thanks for the recognition, but Fast Track is implemented by Peter, not me. It's way above my skillset and patience

[SharraShimada]

5 hours ago, pether said:

Thanks for the recognition, but Fast Track is implemented by Peter, not me. It's way above my skillset and patience

My bad... me and names

 

[EnTro]

8 hours ago, SharraShimada said:

You cant optimize code beyond a certain point whithout compromising stability and compatibility. The Devs need the game to run on every PC. Peter (the mod author) on the other hand can just ignore some problems and be fine with it. He does not need to satisfy paying customers.

 

Please don't take this the wrong way: but is this what you have convinced yourself is the reason for the performance issues? or is this the devs stating that they did all they can but there is no more room? If the community keeps trivializing the performance issue, the devs will never invest in fixing it

I have professionally quite some experience working with, coding for and optimizing scientific simulation software. While ONI does simulate quite some things like fluid flow, heat exchange, pipe flow dynamics, the math is really simple: linear equations on matrix organized data or even simpler linear branched networks with sequential flows. An ONI map is small, just 256 by 384 (SO only slightly bigger) tiles. A decent CPU can churn that like a buttermilk factory churns through milk. (Pathfinding is complex, but a solved issue with the right implementation on such small maps)

The issue with games typically is that 1) game engines aren't meant for these kind of computations, 2) game developers have no experience with high performance calculations, 3) it is seen as a small concern because the game runs well when you start it. Performance gets little focus, implementations are limited by the engine and developers do not have the experience to develop the code bypassing the engine with much more effective solutions.

With a bit of developer focus and the right programmer, the performance thing can be solved. Compatibility and stability would be no concern if implemented correctly, the simulation would be the same, just computed faster. In fact it would improve the experience for players with an older PC.

I'd love to see ONI run well from early to late game with 50 dupes without needing a CERN level supercomputer to break the 25 FPS limit consistency on 1x speed.

[Gurgel]

You are essentially asking for a re-implementation by performance experts. That will not happen. At least not anytime soon. Get better hardware if you want more speed. It is a _game_ after all and games ask for more computing power all the time. 

[pnambic]

1 hour ago, EnTro said:

(Pathfinding is complex, but a solved issue with the right implementation on such small maps)

The entire physics sim, according to the author of the Fast Track mod, is 20% of the performance budget. The issue is pathfinding in an environment that is modified from diverse sources (dupe actions, falling sand, melting/congealing of tiles, automation locking/unlocking doors), and the fact that errand assignment requires pathfinding and ends up about cubic in number of dupes * number of open errands. This is not trivial to parallelize, and even if it were, I've seen several reports that the hardware factor that dominates ONI late game performance is main memory latency. So, while there are very likely some improvements to be made, substantial gains on consumer hardware seem pretty unlikely.

[Gurgel]

If it is memory latency, then yes, not much that can be done on software side or by using more CPUs. Would explain though why the AMD "3D" CPUs with 96MB L3 cache seem to do a lot. 

Well, maybe a complete, cache and memory-burst aware reimplementation could do quite a bit, but that is really hard to do, makes everything very inflexible, takes a lot of expertise and costs a lot of money and time. So, again, not going to happen. Also, on current higher-end gaming CPUs, ONI runs well.  I have an 7800X3D and I have not noticed any lag at 3X. I do typically play classic start, but I do not do excessive dupe numbers. 12...16 in the end-game usually.  

[SharraShimada]

22 hours ago, EnTro said:

Please don't take this the wrong way: but is this what you have convinced yourself is the reason for the performance issues? or is this the devs stating that they did all they can but there is no more room? If the community keeps trivializing the performance issue, the devs will never invest in fixing it

I have professionally quite some experience working with, coding for and optimizing scientific simulation software. While ONI does simulate quite some things like fluid flow, heat exchange, pipe flow dynamics, the math is really simple: linear equations on matrix organized data or even simpler linear branched networks with sequential flows. An ONI map is small, just 256 by 384 (SO only slightly bigger) tiles. A decent CPU can churn that like a buttermilk factory churns through milk. (Pathfinding is complex, but a solved issue with the right implementation on such small maps)

The issue with games typically is that 1) game engines aren't meant for these kind of computations, 2) game developers have no experience with high performance calculations, 3) it is seen as a small concern because the game runs well when you start it. Performance gets little focus, implementations are limited by the engine and developers do not have the experience to develop the code bypassing the engine with much more effective solutions.

With a bit of developer focus and the right programmer, the performance thing can be solved. Compatibility and stability would be no concern if implemented correctly, the simulation would be the same, just computed faster. In fact it would improve the experience for players with an older PC.

I'd love to see ONI run well from early to late game with 50 dupes without needing a CERN level supercomputer to break the 25 FPS limit consistency on 1x speed.

No hard feelings. But your claim IS adressed for YEARS now. The devs are doing what they can to improve, but there is only so much they CAN do, without breaking things. And please STOP calling the devs idiots. They KNOW what they are doing. And no, you are NOT better than they are. And no, you dont need any supercomputer (you dont even know what that is...) My Ryzen 5800X can handle late game colonies just fine.

[blakemw]

On 6/25/2024 at 5:49 PM, Gurgel said:

I do not have any performance issues in ONI since I upgraded to an AMD 7800X3D. I usually play classic-start, so large bases.

Also note that Intel gives a much worse experience at lower FPS than AMD does, due to inferior task-switching, so 30 FPS on AMD may feel more like 60FPS on Intel. 

Well, I want to note that since I upgraded to an Intel i5-12400F I've also not had any performance issues.

Now obviously there's no such thing as a CPU powerful enough to satisfy every user, just as Factorio will run well on a potato but the most mega of megabase builders need an AMD 7800x3D to get the performance they desire.

But in my case, the benchmark I am using is "Can play Spaced Out normally" (e.g. achieve victory conditions). With my old CPU which was some 4th gen intel, the base game was okay but for Spaced Out I had to strictly limit how many planetoids I colonized or performance would collapse. With the new CPU I'm not really thinking about the performance consequences. Like I'm sure if I printed 50 dupes I'd run into problems, but I don't want to print 50 dupes for reasons quite other than performance.

[EnTro]

21 hours ago, pnambic said:

The entire physics sim, according to the author of the Fast Track mod, is 20% of the performance budget. The issue is pathfinding in an environment that is modified from diverse sources (dupe actions, falling sand, melting/congealing of tiles, automation locking/unlocking doors), and the fact that errand assignment requires pathfinding and ends up about cubic in number of dupes * number of open errands. This is not trivial to parallelize, and even if it were, I've seen several reports that the hardware factor that dominates ONI late game performance is main memory latency. So, while there are very likely some improvements to be made, substantial gains on consumer hardware seem pretty unlikely.

Now that is some good feedback, because it is something that can be checked with a quick experiment. My baseline is my game running at 12-14 FPS at 3x speed (or 25ish FPS at 1x speed). So first I set all priorities for all dupes to disallow. There should be no more open errands for my dupes to check or pathfind, other than some personal hangout time. This had NO effect on the FPS. Next I also take the other side out of the equation: the dupes. Going down from 33 dupes to just 2 (both in a rocket, so with minimal pathfinding) only increased the FPS to 14 to 17 FPS.

Clearly the observation is that dupe pathfinding is not the major bottleneck for me, and some other part of the simulation must be.

25 minutes ago, SharraShimada said:

No hard feelings. But your claim IS adressed for YEARS now. The devs are doing what they can to improve, but there is only so much they CAN do, without breaking things. And please STOP calling the devs idiots. They KNOW what they are doing. And no, you are NOT better than they are. And no, you dont need any supercomputer (you dont even know what that is...) My Ryzen 5800X can handle late game colonies just fine.

Please do not get insulted nor insulting.

I absolutely do not call the devs idiots (no idea where you got that from), but I know that experience overlap between a game developer and efficient computing programmer is generally limited. They use game engines to build a game, but game engine are absolutely not optimized for these tasks. They are meant to make Mario jump on a mushroom, play a sound and render graphics using the GPU. Most simulation code that a non-expert wrote can be sped up 3 to 10x times with simple fixes.

I use supercomputer metaphorically. While my 8700K may be several years old, the fastest consumer CPU on the market is only 65% faster on the fastest core, your 5800X just 28%. It is great that your bases run well for you on your PC, but perhaps you have a different playstyle. ONI is a set your own goal kind of game. While 33 dupes is a decent amount, it is feels hardly enough to populate all the asteroids.

Some improvements would help a lot. Even a 50% increase in performance would be a big change. Especially for people with low-end computers that start to feel the game become less responsive in the midgame.

 

15 minutes ago, blakemw said:

Well, I want to note that since I upgraded to an Intel i5-12400F I've also not had any performance issues.

Now obviously there's no such thing as a CPU powerful enough to satisfy every user, just as Factorio will run well on a potato but the most mega of megabase builders need an AMD 7800x3D to get the performance they desire.

But in my case, the benchmark I am using is "Can play Spaced Out normally" (e.g. achieve victory conditions). With my old CPU which was some 4th gen intel, the base game was okay but for Spaced Out I had to strictly limit how many planetoids I colonized or performance would collapse. With the new CPU I'm not really thinking about the performance consequences. Like I'm sure if I printed 50 dupes I'd run into problems, but I don't want to print 50 dupes for reasons quite other than performance.

That is the thing, I have a decent CPU (top of the line when ONI was released 7 years ago), and I can hardly run a 33 dupe base on 3 planetoids with 4 rockets in SO. I don't think I'm doing anything excessive in the context of the game, yet I see my FPS crawl to a slideshow. This can't be right.

[pnambic]

12 hours ago, EnTro said:

Now that is some good feedback, because it is something that can be checked with a quick experiment. My baseline is my game running at 12-14 FPS at 3x speed (or 25ish FPS at 1x speed). So first I set all priorities for all dupes to disallow. There should be no more open errands for my dupes to check or pathfind, other than some personal hangout time. This had NO effect on the FPS. Next I also take the other side out of the equation: the dupes. Going down from 33 dupes to just 2 (both in a rocket, so with minimal pathfinding) only increased the FPS to 14 to 17 FPS.

Clearly the observation is that dupe pathfinding is not the major bottleneck for me, and some other part of the simulation must be.

That's super interesting! Can I have your save to run some experiments on that?

For comparison: I just took my current base - cycle 1650, Classic start, 28 dupes, all planetoids revealed) and purged all dupes on the home base, going from 28 to 5 (one in a rocket, 4 on the teleroid). My FPS went from ~22 to ~32 on default speed, with the camera in the exact same spot. Which is about what I'd expect.

Also, I generated a new Quagmiris start, dropped into debug, and revealed all planetoids completely. That didn't move FPS in a noticeable way (55-56 when looking at the printing pod and the three idle dupes there, again default speed).

This is on the current release build, running on a slightly moldy potato (i7-6700, 16GB RAM, GTX970). Fast Track not installed, and only one mod that is known to be a performance drain (pipe flow overlay, with the overlay switched off). Notably, I do not run the multi-color gas-overlay mod that replaces the breathability overlay on F1, that has completely killed performance for me in the past.

[Gurgel]

14 hours ago, blakemw said:

Well, I want to note that since I upgraded to an Intel i5-12400F I've also not had any performance issues.

Now obviously there's no such thing as a CPU powerful enough to satisfy every user, just as Factorio will run well on a potato but the most mega of megabase builders need an AMD 7800x3D to get the performance they desire.

But in my case, the benchmark I am using is "Can play Spaced Out normally" (e.g. achieve victory conditions). With my old CPU which was some 4th gen intel, the base game was okay but for Spaced Out I had to strictly limit how many planetoids I colonized or performance would collapse. With the new CPU I'm not really thinking about the performance consequences. Like I'm sure if I printed 50 dupes I'd run into problems, but I don't want to print 50 dupes for reasons quite other than performance.

The i5-12400F has "only" regular caches, but both it and the 7800x3D have a lot faster main memory access than older CPUs. Most software is CPU limited, not memory-speed limited, so in other programs, the difference may be a lot less.

If you are limited by main memory speed, however, as ONI seems to be with larger bases, larger caches help a bit, but more CPU cores do not help at all and using them may actually slow things down. Hence you need that memory bandwidth and that means modern RAM and modern CPUs. Yes, I am well aware that, for example, 7 year old CPUs may well perform fine for a lot of things or even most things. But ONI is not one of them. 

 

[EnTro]

4 hours ago, pnambic said:

That's super interesting! Can I have your save to run some experiments on that?

For comparison: I just took my current base - cycle 1650, Classic start, 28 dupes, all planetoids revealed) and purged all dupes on the home base, going from 28 to 5 (one in a rocket, 4 on the teleroid). My FPS went from ~22 to ~32 on default speed, with the camera in the exact same spot. Which is about what I'd expect.

Also, I generated a new Quagmiris start, dropped into debug, and revealed all planetoids completely. That didn't move FPS in a noticeable way (55-56 when looking at the printing pod and the three idle dupes there, again default speed).

This is on the current release build, running on a slightly moldy potato (i7-6700, 16GB RAM, GTX970). Fast Track not installed, and only one mod that is known to be a performance drain (pipe flow overlay, with the overlay switched off). Notably, I do not run the multi-color gas-overlay mod that replaces the breathability overlay on F1, that has completely killed performance for me in the past.

Sure! I will do that when I get back to my PC with ONI. Be warned: my base build is a bit messy. It is my first playthrough in years and I had to learn and adapt on the fly. I was running the new DLC beta, but I did not notice a significant difference in FPS between the beta and the normal release. I think nothing was changed.

 

1 hour ago, Gurgel said:

The i5-12400F has "only" regular caches, but both it and the 7800x3D have a lot faster main memory access than older CPUs. Most software is CPU limited, not memory-speed limited, so in other programs, the difference may be a lot less.

If you are limited by main memory speed, however, as ONI seems to be with larger bases, larger caches help a bit, but more CPU cores do not help at all and using them may actually slow things down. Hence you need that memory bandwidth and that means modern RAM and modern CPUs. Yes, I am well aware that, for example, 7 year old CPUs may well perform fine for a lot of things or even most things. But ONI is not one of them. 

 

The biggest map, Spaced out is just 256 x 384 tiles in size. That makes the core map data needed for simulations a few MB. A half decent DDR4 setup can read/write that thousands of times per second. ONI only requires a tick rate of 5 per second. If memory-speed is the problem, it's just a symptom of the performance problems with the code.

ONI is 7 years old and hasn't been updated that much in terms of simulation complexity. A 7 year old, then top-of-the-line CPU with a good system should have no issues running ONI. On the steam forum a user with a 14900K reported having performance issues too. It's not just old hardware

[EnTro]

So I have uploaded the save game here: https://filebin.net/rhmmnrl7fjsc1dtm

[pnambic]

1 hour ago, EnTro said:

So I have uploaded the save game here: https://filebin.net/rhmmnrl7fjsc1dtm

Thanks; got it to run, at about the FPS counts you mentioned, so our respective potatoes seem halfway evenly matched. I'll report back within 24 hours.

(surprised to see no mods whatsoever, not even "bigger camera zoom out". Or did I screw up something there?)

[EnTro]

11 minutes ago, pnambic said:

Thanks; got it to run, at about the FPS counts you mentioned, so our respective potatoes seem halfway evenly matched. I'll report back within 24 hours.

(surprised to see no mods whatsoever, not even "bigger camera zoom out". Or did I screw up something there?)

Nope, no mods. All vanilla, just the Spaced Out DLC. Enjoy my potato base

[Gurgel]

15 hours ago, EnTro said:

The biggest map, Spaced out is just 256 x 384 tiles in size. That makes the core map data needed for simulations a few MB. A half decent DDR4 setup can read/write that thousands of times per second. ONI only requires a tick rate of 5 per second. If memory-speed is the problem, it's just a symptom of the performance problems with the code.

ONI is 7 years old and hasn't been updated that much in terms of simulation complexity. A 7 year old, then top-of-the-line CPU with a good system should have no issues running ONI. On the steam forum a user with a 14900K reported having performance issues too. It's not just old hardware

Your estimations are really meaningless. You seem to assume everybody codes things in assembler by hand. 

[NeoDeusMachina]

11 hours ago, pnambic said:

Thanks; got it to run, at about the FPS counts you mentioned, so our respective potatoes seem halfway evenly matched. I'll report back within 24 hours.

(surprised to see no mods whatsoever, not even "bigger camera zoom out". Or did I screw up something there?)

Alt+s does the same thing, I never understood why that mod was necessary. Does it do anything else then allow you to zoom out more?

[pnambic]

2 hours ago, NeoDeusMachina said:

Alt+s does the same thing, I never understood why that mod was necessary. Does it do anything else then allow you to zoom out more?

It's just more seamless. With Alt-S you switch off the UI, zoom out, hit Alt-S again to get the UI back, then you can move around and play at that zoom level, but any change to zoom snaps you back into the original bounds. The mod just lets you zoom to where you want, when and how you want.

[NeoDeusMachina]

1 hour ago, pnambic said:

It's just more seamless. With Alt-S you switch off the UI, zoom out, hit Alt-S again to get the UI back, then you can move around and play at that zoom level, but any change to zoom snaps you back into the original bounds. The mod just lets you zoom to where you want, when and how you want.

Makes sense, thanks