Vanishing Steam Bug - Bug Report Requested

I'll start by explaining the life cycle of a liquid particle in this scenario. Ask if anything is unclear.

Quote

1. A liquid particle is emitted from a Liquid Vent. The particle is purely visual and does not interact with the sim. In fact the liquid is removed from the sim while it's falling and re-added when it lands.

2. The particle falls and determines when it should re-add itself by checking the current cell it's in.
    - if cell = gas: continue falling
    - if cell = vacuum: continue falling
    - if cell = liquid: 
      - if cell's element = particle's element:
        - if not much liquid: continue falling
        - otherwise: add to sim
      - if different element:
        - if particle's molar mass > cell's molar mass: continue falling
        - otherwise: add it to the sim
    - if cell = solid: add to sim the above cell
        
3. Now the sim takes over adding the contents to the cell. There are three parts to it.

  a. If cell is the same element simply combine them.

  b. If the cell is solid (uncommon case) try to find an orthogonally adjacent liquid/gas/vacuum cell to use instead for part c. 

  c. Try to displace the contents of the cell.
    - if cell = vacuum: it already is displaced
    - if cell = gas: try to displace
    - if cell = gas and failed to displace AND the cell above is liquid: swap contents with the above cell (bubble up gasses)
    - if cell = liquid: try to displace
    
    If after trying all of the above the contents cannot be displaced we still need to place the liquid somewhere. Ideally we would try recursively displacing elements further away but that would have performance implications so instead we handle this by deleting the cell's mass.

      example 1: cell is 3kg oil, liquid is 1kg water. Cell becomes 2kg oil
      example 2: cell is 1kg oil, liquid is 3kg water. Cell becomes 2kg water
      example 3: cell is 2kg oil, liquid is 2kg water. Cell becomes vacuum.

    This is what is happening here.

In your example of a water-steam turbine setup this is unlikely to occur every tick in the worst case scenario. I could happen in @sheaker's case with oil in the steam chamber.

In reading over the liquid displacement code I noticed we stop displacing liquid when it's below a minimum liquid mass (0.01 kg) even when it's possible to distribute it into adjacent cells. I plan to change this so it will try to distribute everything before deleting that tiny amount. However, this would account for at most 30 kg loss in a cycle if every tick it occurs and you're seeing well over that.

There two other cases where mass deletion occurs:

1. Gas cell is less than 1 microgram (0.000000001 kg)

2. Gas cell is less than 1 gram (0.001 kg) AND all it's orthogonally adjacent neighbors have more than 1 kg mass. This is to help remove stray amounts of gas.

Once we figure out a way to somewhat reliably reproduce this bug we can examine whether any of the above is relevant. It could also be an issue with saving/loading the falling water but it sounds like you tried that.

Have you seen this happen when the steam turbine is not disabled?

1 hour ago, EricKlei said:

the life cycle of a liquid particle

The pseudo code you provided fits exactly in with my understanding of the game, including all the deletion scenarios you described. Your description confirmed what I have believed. 

The fact that different amounts are deleted by loading the same save file different times, tells me that the issue is related to save/load. I don't think it's save/load of the falling water, rather something related to save/load on a steam turbine. Even when @sheaker produced an example that was claimed to not be related to save load, I'm not convinced.  What I saw in his video example was a turbine that was stuck in the "off" position upon game load.  After letting the game run for a while, he then connected the turbine to automation, upon which the bug started deleting massive amounts of steam. 

Does it always require automation?  I have some examples from a year ago that suggest it might not, but those examples also had stuff dropping into crude oil.  I'll try to replicate the mass loss without automation. 

I have a guess that the bug has to do with the steam turbine itself.  I've got an idea to test this, and will share soon. 

2 hours ago, EricKlei said:

It could also be an issue with saving/loading the falling water but it sounds like you tried that.

If it were just save load on falling water then the mass loss would not persist through an entire cycle. However if the steam turbine itself got stuck via automation in some weird state then maybe that would be it. The liquid pipes claim to carry the full 2kg water. Could this be a lie? I can reliably (high probability with varying amounts lost) reproduce steam deletion for steam turbines attached to automation on save/load.  Some combination of these 3 things is the issue.

I'll test a pump as well and see if I can reproduce matter deletion with just a liquid pump attached to automation, 95C water, and a hot room.

@ghkbrew, @Zarquan, @TrippleM any of you up for helping?

Edited April 24, 2021 by mathmanican

If it was an issue with saving I would expect it to happen each time you load the save. It could have something to do with loading the steam turbine and how it interacts with the sim. Does it only happen immediately after a load?

I took a quick peak at the steam turbine code and was going to suggest testing a liquid pump since it has a lot in common but you already thought to try that!

When the steam turbine is operational it sends a message to the sim to consume steam from the cells below. This is handled on the next tick and then the steam turbine receives a callback with the amount consumed. When the generator code runs it consumes the stored steam and outputs water. I'm wondering if the callback is never received when the bug happens. Next time you are able to reproduce it try piping the water into another area to see if that's the case.

  In my opinion it is related to liquid vent or anything that happens after water leaving liquid vent. We can clearly see the water is coming from the steam turbine 2kg/s constantly so inside the pipes there is no water lost. It would be strange to see steam turbine sucking additional portion of steam.

In this post:

Mathmanican said:

On 6/26/2020 at 2:52 AM, mathmanican said:

I increased the mass of crude oil to 400kg and the problem completely stopped. I cannot get steam deletion now with any kind of save/load (fresh restart, or reload).  I wonder if the issue either has to do with viscosity (crude =50) or min flow (100g).

And I think this may be important to remind.

 

Based on this video:

On 6/28/2020 at 9:09 AM, sheaker said:

Ok. I found that this is NOT just a save/load issue. Here it happened not straight after loading the game:

 

We can observe that this is actually not straightforward save/load issue.

Finally in this post:

@DaClown compared mass deletion to heat deletion.

 

I wish I could help You more with this bug.

8 hours ago, sheaker said:

We can clearly see the water is coming from the steam turbine 2kg/s constantly so inside the pipes there is no water lost.

One current hypothesis is that this may be a visual lie.  I'm testing that now, or at least hoping to test it. 

8 hours ago, sheaker said:

Mathmanican said

Further down in the same report, I refuted the comment with, "Well, that theory (about 5*max viscosity) is debunked."  The mass of liquid is irrelevant.

8 hours ago, sheaker said:

Based on this video

Your video shows you loading a game with a steam turbine stuck in the "OFF" position from automation. My current hypothesis is that the bug requires automation to trigger upon save/load.  In other words, the bug hit the second you started the game, and the steam turbine got stuck in some weird state that allows it to delete mass, but only once you start the turbine up (via automation, which you do). 

Your video fits along the hypothesis above. It also matches with my own experimentations.  

The comments from @DaClown about the comparison don't quite fit here (IMO). Liquid flow does not occur if a blob of liquid would be so small that it would get deleted. 

I am currently testing 13 turbines over and over again on save/load, with all but 2 not attached to automation. The one attached to the timer regulary looses quite a bit of steam.

The 11 turbines that are automation free have never lost steam. I'm wondering if a simple fix to this steam loss problem for turbines is to turn them off with doors underneath (so automate something else), in the event you want the turbine to turn off.  This is still a guess. 

Edited April 24, 2021 by mathmanican

Here's an update on failed attempts, with a new save. 

• Tunnel.sav 

I built a new array of 10 turbines (the image in my previous post). The three turbines at the bottom are the original 3, and I regularly see steam deletion in the bottom right of these three turbines, with almost every load (though amount varies). Let's call this turbine the bugged turbine. 

• I have been unable to get steam deletion to happen in any of the 10 top rooms, with or without automation.
• I use bridges to get the steam from the turbine to the vent in the bottom 5 turbines (matching bugged turbine). The top 5 turbines take liquid directly to the vent without a bridge.  So far I have been unable to get steam deletion to occur in any of the top 10 turbines. 

The top 10 turbines I built at the start of cycle 4.  I attached them to the same timer that controls the bugged turbine at the start of cycle 5, and then let the game run a bit to give the turbines a chance to sync up with the bugged turbine. I used an automation bridge to connect things, so I'll probably try direct connections next (but that will probably fail to produce the bug). The bugged turbine probably saw changes to its automation over the course of several cycles (a few days ago and I can't remember every construction detail/change), so maybe the problem happens somewhere when automation is removed and/or changed, or attached to something that is not as regular as a timer. 

@EricKlei, at this point, the only thing I can help with (aside from failed attempts to reproduce the bug on a large scale) is the save file above.  Of the 13 turbines in that file, I only ever see steam deletion in last bottom right one.  If it has to do with internal settings on a turbine, then that bugged turbine will hopefully provide what you need. 

One last fact.  I normally play on Linux.  However, I have a windows computer as well.  The bug happens on the Windows machine.  Both are just 8GB RAM, so perhaps it's a processing speed thing. 

I would love to have someone with a 64GB RAM machine test this save.  If you are interested, then load the file and open it several times. After 1/2 a cycle, delete the timer and let things settle in the turbine rooms.  The middle cell below the turbine will have very different values (ranging from 11kg to 18.4kg).  The quantity of lost mass is not consistent, and doesn't always happen (the 18.4).   The first time I loaded the save on my Windows computer, I saw no mass loss.  It was only after I reopened the save that I saw the mass loss. 

Edited April 24, 2021 by mathmanican

2 hours ago, mathmanican said:

One last fact.  I normally play on Linux.  However, I have a windows computer as well.  The bug happens on the Windows machine.  Both are just 8GB RAM, so perhaps it's a processing speed thing. 

I also play on linux. In my case, I've had steam vanish with on a R5 3600 running with 16GB DDR4 (2x 8GB ECC-UDIMM).

I recently downgraded and am using an intel 4700k with 16GB DDR3 installed. (4x4GB non-ecc). It also happens on the latter.

Issue happened on both Ubuntu and Pop!_OS

My save file, rounded up to 60MB in size will take up a good chunk of the 16GB and is good for hours of non-stop play. But it will inevitably gum up the memory and the OS will take over. Full stop.

4 hours ago, mathmanican said:

I'm wondering if a simple fix to this steam loss problem for turbines is to turn them off with doors underneath (so automate something else), in the event you want the turbine to turn off.

Sorry to say that the first time it happened to me was on a setup that has 100% uptime (and it continues to ping from time to time).

The vanilla map I'm playing on has various ST setups and varying levels of uptime. There is one that hasn't had any loss event so far and it's intermittently active. (This is out of a possible three that are unscathed by the issue - meaning I'm not sure of the second and the third is still under observation - i.e.: not old enough )

The other three have needed refilling.

The map is old enough that it started running into the out of tags issue so the mod for that was also installed.

Did steam vanish before using mods or any other kind of alteration (sandbox,debug ...): Yes. (I state this just in case.)

Is the map currently usable on the latest version of the game: Yes (On hiatus cos' I don't know what I want to build next)

I have some speculative ideas about what can be going on but for now I'll just save them as ramblings of crazies as to not digress.

 

 

 

5 minutes ago, JRup said:

I've had steam vanish with on a R5 3600 running with 16GB DDR4

Thanks for confirming that it is probably not a RAM issue. I have it happen on both my Linux and Windows machine. 

6 minutes ago, JRup said:

the first time it happened to me was on a setup that has 100% uptime

Did the setup have an automation wire connected to the turbine?  It may be on 100% of the time, but did you connect it to automation to pick when to turn it on? Is there any chance at all that an automation wire ever touched the steam turbine port? 

I decided to play around with automation to see if I could trigger the bug.  BINGO.  I got all 10 new turbines to trigger it (though it took 2 loads, so still not 100% predictable).  I deleted the timer, put in a manual switch, flicked it a few times off/on and then put a timer back on it (I let 1/2 cycle pass).  Then I saved the game and reloaded. 

• Tunnel2.sav - Before Bug kicks in (though I think the top 5 turbines already lost a very small amount). 
• Tunnel3.sav - Near end of 5th cycle, the bug has gobble up plenty of steam from all 10 turbine rooms. 

The top 5 turbine rooms all lost the same amount. The bottom 5 turbine rooms all lost the same amount. This suggests it isn't random, but rather depends on something else.  What that is, not sure. 

• The design feature difference between the two rows of turbines is the location of a bridge between the outlet of the turbine and liquid vent.  I don't think the bridge is the issue. 
• If the event were random, then I would be ready to suspect a gas-liquid interaction at the vent.  This is because gas movement includes a random event.  Because the results seem identical on each group of turbines, I suspect the issue is NOT related to the physics simulation, and rather related to something else.

Why do the two groups of turbines have differing results?  If it's automation related, here is one possible answer.  I connected the turbines to automation by laying wire left to right. Perhaps, upon laying the wire an initially triggering an "off" condition, some internal setting in the turbine was changed.  I then later connected each group of turbines to the timer automation with a bridge.  So each group of 5 turbines was attached to the timer a the same instant (which could be where the internal setting was triggered). 

21 minutes ago, JRup said:

I have some speculative ideas about what can be going on but for now I'll just save them as ramblings of crazies as to not digress.

At this point, I think we need the "ramblings of crazies" to track this down.  Basically, this entire bug report and my work above is the ramblings of a crazy.  

Right now my best guess is that the steam turbine itself is the problem, and that each time automation is set/reset by building/modifying current automation, something changes in the turbine. You can see, when laying liquid pipes, that the simulation does something anytime you adjust pipes.  Does something similar happen on the automation end, anytime you adjust a wire that touches the turbine. 

The bug could easily be related to the turbine itself without automation being the culprit, rather we can just speed up how long it takes for the bug to become noticeable when we adjust automation wires/buildings attached to the turbine. 

That's enough crazy ramblings from me for now.  I'll ramble just a bit more, as I mention things I'm going to try at some point. 

• Pass the output of a turbine through a liquid storage tank (with and without a buffer) to see if the problem occurs in the pipes between turbine and vent. 
• Design a room where the 95C liquid vents into 95C liquid, and then goes through a spillway to return to steam (to test if the issue is steam/gas interaction). 
• Look for mods, or make one, that shows more details about steam turbines, such as how much liquid is currently sitting in them, and then see if I can connect the current contents of the turbine to bug. My guess is that this is the most likely culprit. The turbine's internal buffer somehow gets out of sync with the actual game. This one sounds much simpler for a developer to check. It's the idea that I think is most likely the problem. But remember, these are just madman ramblings. 
• See if I can replicate the bug using gas pumps and a liquid pump.  If it's an automation bug issue that affects more than just turbines, then I should be able to get a liquid pump's buffer (which I can see) to produce the bug. I haven't seen any posts on this anywhere, but I also don't think people use automation on the intake pump very often, as you can easily control the output and just let the pump be blocked.
• Add a liquid shutoff on the outgoing side of an already bugged turbine to see if blocking the out flow fixes an internal buffer problem, as well as preventing automation from ever needing to be attached to the turbine.  If I can find a way to fix a problem, then while it may not prove the source cause, it can provide relief (treat symptom not cause). 

If crazy ramblings are being solicited...

The talk about automation brings to mind the power shutoff and mechanized door bugs. I think the shutoff bug is well known. My understanding is that saving and then reloading on a particular frame leaves the shutoff in a bugged state. Or something like that, I never used them and never looked into the bug very much.

The door bug is when a solid cell of the door exists even though the door is open. Typically seen in boilers where a door is used to inject heat in a controlled manner. The door gets bugged out and the boiler overheats because the "open" door is still letting heat through. I've only had it happen to me once and didn't dig into it any. My speculation is that it's tied to rapidly cycling the door automation.

Neither of these bugs is a smoking gun for the steam issue, rather just examples of where the automation timings cause problems.

 

I don't have the DLC nor will I be getting it so I can't load the saves posted. And I can't reproduce it myself. That being said, I think trying to pin the bug down to either the turbine or the water/steam interaction would be helpful. Instead of reboiling the turbine water, what happens if it is just collected? Does 1000 kg of steam always result in 1000 kg of water? If not, that clearly implicates the turbine itself.

As for the water/steam interaction, I could see how slight timing differences could be a culprit. Water leaves a vent per second but the falling particle moves per frame. Usually the reconstituted water behaves properly, but is there a particular frame where things don't behave? If so, whether or not the bug occurs could depend on stuff like game speed, frame rate, distance falling, etc. The idea of a spillway seems useful as it may convert from unreliable frames to robust ticks. Might the old mesh tile under a vent trick serve the same purpose? Or how about partial evaporation?

Edited April 25, 2021 by wachunga

2 minutes ago, wachunga said:

can't load the saves posted

I can reproduce the bugged turbines in a non DLC save for you to experiment with. I'll have to much later today. 

4 minutes ago, wachunga said:

If crazy ramblings are being solicited...

Very much wanted.

3 minutes ago, mathmanican said:

I can reproduce the bugged turbines in a non DLC save for you to experiment with.

By all means. I'm happy to bash my head against a wall for a bit if it helps you out.

1 hour ago, mathmanican said:

Thanks for confirming that it is probably not a RAM issue. I have it happen on both my Linux and Windows machine.

Don't forget it was ECC capable memory. Though I have my qualms about the rig actually taking advantage of ECC given that it is an unofficially supported feature from AMD that varies from one motherboard manufacturer to another, but it was detected in the OS (asus CH VI)

I just don't feel comfortable without ECC and given the fact that the game data is in ram when playing is another strong need for it. (That we do without ECC is miraculous in itself but it's just like driving without seat-belts.)

I've never had the game crash when running out of memory on the DDR4-ECC setup. The game would endlessly hitch until I could access the save menu... Or just right out stop working and would have to kill it from the OS monitor - or hard reset the whole computer.

On this DDR3 box, it dies.

 

1 hour ago, mathmanican said:

Did the setup have an automation wire connected to the turbine?  It may be on 100% of the time, but did you connect it to automation to pick when to turn it on? Is there any chance at all that an automation wire ever touched the steam turbine port?

Steam vanishing happened before setting up automation. This particular setup is used to drain magma from the core so 100% uptime was needed.

Vanishing events were happening often enough that automation was put into the system for maintenance (I eventually had the idea to connect the refill permanently - the more you play the more you learn, I think)

1 hour ago, mathmanican said:

The bug could easily be related to the turbine itself without automation being the culprit, rather we can just speed up how long it takes for the bug to become noticeable when we adjust automation wires/buildings attached to the turbine. 

All other setups are automated to activate in the 200ºC range.

1 hour ago, mathmanican said:

At this point, I think we need the "ramblings of crazies" to track this down.  Basically, this entire bug report and my work above is the ramblings of a crazy.  

Ok. I will add some spice to the pot.

Besides, the fact that it happens enough for people to alter their build style means we're not hallucinating.

Time to bark at the moon, i.e.: crazy talk on my part:

I believe it is not only one set of circumstances that makes steam disappear.

Silly things happen on load. This will happen all the time and is "completely normal phenomenon":

• Duplicant traversing magma in atmo suit: unjustly receives heat stroke. Internal timer was artificially fast forwarded to receive debuff.
• Duplicant surrounded by expecting poke-mommies: receives full brunt of crab-inism and gets incapacitated. Exposition time to mommies was artificially compounded and dupe was lovingly clobbered.
• As @wachunga reminds us, some things are affected by saving on certain times - power shutoff/mechanized airlock.

For the on-load version of the issue:

If the ST is off, nothing happens. We're safe.

If the ST is frozen mid toggling to on then trouble hits. Steam extraction starts, gets compounded and isn't reported so this mass is effectively lost.

 

But how is it that steam vanishing happens mid game?

 

For the mid game version of the issue:

I can't help but feel that at some point the X/Y coordinates of the steam turbine get somehow altered in memory at some point...

The ST is still where it should, the intakes are working fine and subtracting the amount of steam from the chamber appropriately but internal reporting is saying it's receiving less than what's extracted thus returning less than what's taken. Pipes are ok.

Story time: On one recent crash I had prior to adding a mod to ameliorate the out of tags issue in vanilla I had something odd happen with a totally unrelated material: regolith.

I'd built a run of the mill C-miner setup and was stacking all the regolith on a single tile. Oddly enough, after some hours of play, game started acting up a bit and this time it hit the regolith stacks. They all appeared further to the left of where they should be. The game later crashed. :/

After that I installed the mod for out of tags. I'll admit it's an old map already after 5k+ cycles and the molten core hasn't run out of magma yet. There's more igneous rock than I can effectively use, lol.

I have another bug related to steam turbines that I might be posting soon: copy paste a steam turbine in debug mode and it gets shifted to the right by one tile...

 

 

 

 

7 minutes ago, JRup said:

If the ST is off, nothing happens. We're safe.

All of my experiments have the turbine in OFF mode via automation in the middle of a 10s toggle timer.  So effectively the turbine is off for several seconds on load.

Thanks for the ramblings. The one tile off bug could be related. Save/load has so many issues (atmosuits seem to not protect at all on load as you will get sopping wet debuff as well as the scalding you mentioned).

You've all given ideas for more tests. It's gonna be a fun weekend.

18 minutes ago, JRup said:

I have another bug related to steam turbines that I might be posting soon: copy paste a steam turbine in debug mode and it gets shifted to the right by one tile...

Can confirm. A thought was if copy pasting the bugged turbine with the selection tool retained the buggedness in the new turbine. Wouldn't say much but could help in making more bugged turbines to test. Assuming the copy paste process didn't introduce it's own bugs.

And two other weird timing issues since we are rambling. Cycled tepidizers to ignore max temp and rapidly cycled stuff to not consume power. I haven't used either so I'm no expert. I think the tepidizer is particularly weird in that save/loading further changes the behavior.

Edited April 24, 2021 by wachunga

1 minute ago, wachunga said:

A thought was if copy pasting the bugged turbine with the selection tool retained the buggedness in the new turbine.

If you want an additional piece of mind... Get your tinfoil hat.

Why do some steam turbines behave while others are fine as is? It might not be the turbine's fault (at least not completely).

I'm mostly suspecting the map. Yes. The. Map.

Not all the map has the same characteristics, this is seen in the backdrop of the biomes even. But I still can't quite put a finger on it to sufficiently describe it.

For example: I recently found out that geyser stats such as output and dormancy are generated with the coordinates it's spawned at. (Just try to paste one in the same spot, I got the same thing 10/10 times.)

So is there some noise in the randomness of movement of gas/liquid dependent on any given X/Y coordinate?

I've said too much.

17 minutes ago, JRup said:

Get your tinfoil hat.

Done. I love it.

4 hours ago, wachunga said:

Instead of reboiling the turbine water, what happens if it is just collected?

The top row of 5 turbines triggered the bug after one load. I captured the liquid from the right most turbine in a storage tank.  While the 4 steam rooms to the left lost 25% of their mass, there was zero mass loss (up to turbine internal buffer) for the right most turbine. 

-> This suggests it's the physics sim, not the turbine.

Gonna work on getting you a save with bugged turbines that isn't on the DLC. 

Edited April 25, 2021 by mathmanican

@mathmanican, So I just sat and reloaded your "Tunnel Cycle 5.sav"  for the past 2.5hrs.  I've seen the mass deletion in the rightmost many times. 

Some observations:

• I've been able to trigger the bug most consistently after restarting ONI.  I don't think I was able to trigger it once by reloading the map without first quitting to desktop.
• When the deletion happened it began immediately after load.  Though I didn't run the save for long when it didn't happen so I'm biasing observations.
• The deletion occurs steadily.  A few times I was able to use the debug tool to measure the mass of the steam while the deletion occurred.  By watching the maximum amount of steam during the off phase of the timer, I was able to observe a steady drop in total steam while the bug was occurring, in the range of 2-8kg per 20s cycle of the timer.
• The deletion seems to happen in 2kg increments, most frequently (always?). Again this is based on watching the deletion happen with the debug tool.
• The deletion seems to continue until I do something.  If I just let it run (don't move the mouse, or press keys) it will continue to less than 10kg/tile.  I should probably go back and see how low it will go.
• I'm not positive (given the random nature of this), but I think "observing" the turbine's exhaust pipe causes the bug to stop.  Simply opening the piping overlay (pressing F6) consistently caused the bug to stop.  I think mousing over the pipes made it stop too.

My best method of triggering the bug is.  Opening ONI from scratch. Loading your save file.  while the loading screen is up, positioning my mouse in the location of the center steam tile (this was right under the "oa" in "Loading" on my computer).  Immediately after the map loads, while everything is still grayed out press the spacebar to unpause the simulation immediately.  I then observed the mass in the center tile.  It will fluctuate as the turbine turns on and off.  If the bug does not happen it will reach 19.1 or 19.2kg briefly right at the end of each off phase.  This number starts dropping by a few 1/10ths of a kg each timer cycle if the bug is occurring.

My interpretation is that this is a problem in the piping.  Either when the turbine tries to output into the pipe or possibly something to do with the bridge or vent.

55 minutes ago, mathmanican said:

The top row of 5 turbines triggered the bug after one load. I captured the liquid from the right most turbine in a storage tank.  While the 4 steam rooms to the left lost 25% of their mass, there was zero mass loss (up to turbine internal buffer) for the right most turbine.

If it is a piping issue, changing the piping to capture the output of one turbine may have prevented the bug for that turbine.

45 minutes ago, ghkbrew said:

changing the piping to capture the output of one turbine may have prevented the bug for that turbine

Agreed that this could be one possibility.  I've thought about it being a piping issue as well.  At some point, I'll run some tests with just pumps and no turbines, to see if I can get it to happen. 

@wachunga, I made a save for you to play with.  This one only has 10 turbines, as well as some fun experiments with bubble physics from @Qwertymenow's recent post (I wanted to radically increase the bubble creation rate, and noticed it was using similar principles as a bypass pump).  Here is a save that you should be able to open without the DLC. 

TestingGround Cycle 14.sav

You may have to load the game several times to view the bug in action.  I observed over 30kg of steam deletion from each chamber (average mass dropped from 19.9kg to 18.8kg) before I decided this save would work. 

To create this, here are the steps I followed:

1. Create everything, fill pipes with liquid, leave aquatuners in off position. 
2. Paint in 20kg/tile of steam to each turbine, and turn on aquatners. 
3. After turbines are up and running, connect them to a manual automation switch.
4. Flick automation switch off/on many times while game is running. 
5. Delete manual switch, and replace with timer.  Let run for a bit. 
6. Delete timer, and swap back to manual swith.  Flick it many times over 30 seconds or so. 
7. Replace manual switch with timer. Let run, and then save game with timer in off state (not sure this part matters). 

From there, I just reload the save multiple times. 

54 minutes ago, ghkbrew said:

I've been able to trigger the bug most consistently after restarting ONI.

I get the bug to appear quite often on a reload while game is open. I don't have to restart ONI at all.  Fun. This is the frustrating part, namely that reproducibility is tough. 

The fact that I have now been able, on 3 different maps, recreate the bug so that it occurs on multiple turbines all at once, tells me that it probably isn't the map (gonna put the tinfioil cap away for a bit @JRup).  

In the save I sent Wachunga, I connected all 10 turbines together before I started messing with automation toggles.  The 10 turbines seem to produce the bug at the same rate. 

57 minutes ago, ghkbrew said:

The deletion occurs steadily

Yep.  I agree. I'll check the other things you observed as well, to confirm them. Some are things I've noticed as well, but not really sure what they mean.  A piping issue with how the steam turbine puts liquid into the pipes could be it, however, I originally built all 10 turbines with a direct connection.  I only later swapped 5 of the turbines to use bridges.  Unfortunately, the bug started happening AFTER this change occurred (as it was then that I started fiddling around with constructing/deconstructing automation toggles). Next time I open the game, I'll deconstruct and reconstruct a pipe segment, and then save load the game, to see if that fixes the bug.  That is what I did in my liquid tank experiment (I save/loaded before running things). 

I think we have plenty here for @EricKlei and @klei.ruby to work with (I hope).  Lots of saves that are messed up, and hopefully a way to reliably produce more turbines that are messed up. 

9 minutes ago, mathmanican said:

The fact that I have now been able, on 3 different maps, recreate the bug so that it occurs on multiple turbines all at once, tells me that it probably isn't the map (gonna put the tinfioil cap away for a bit @JRup).  

No biggie, as long as we're having fun. Besides, we're trying to track something that has caused many a migraine. Hopefully, there's enough info for the devs to go full Ahab on this one (finally killing it, perhaps).

1 hour ago, ghkbrew said:

I think "observing"

So we have an ONI Uncertainty Principle. 

I will test this again soon. I'm pretty sure on my slow potato computer I have been able to observe the pipes in the pipe overlay and still watch steam drop.

3 minutes ago, mathmanican said:

So we have an ONI Uncertainty Principle. 

Clearly it's a Heisenbug

8 hours ago, mathmanican said:

Here is a save that you should be able to open without the DLC. 

Popping in to confirm that this save demonstrates steam deletion for me in all 10 turbines. After one cycle, steam was reduced from ~20 kg per tile to ~10 kg per tile. Roughly the same for all 10 turbines. Had to load the save twice for the deletion to occur. First load no deletion. Second load from in game, didn't quit out or anything.

Edited April 25, 2021 by wachunga

Update on some things I've found.

Starting from the "TestingGround cycle 14" save @mathmanican posted just above, I simplified it in the interest of removing potential hiccups and extraneous factors. Apologies to those looking at the DLC saves. This shows steam deletion pretty reliably for me, takes just 1/8 of a cycle to see it happening or not.

test3.sav

 

I'll let it run until I can see deletion. Then if I pause and frame advance 15 frames and unpause, the deletion stops. I've tried less than 15 and deletion continues, every timed I've done 15 or more the deletion stops.

Starting from a fresh load, again I'll let it run until deletion is verified. Then replace the bottom row of 650K thermium with 600K. Once the steam temp drops after a few moments, deletion stops. Restore the 650K thermium and deletion starts again once the steam warms up.

I've done a few tests with changing the piping resulting in both the deletion continuing or stopping. No conclusions but I suspect fiddling with the piping sometimes does the same thing as advancing frames, essentially bringing the flow into a stable state.

 

Edited April 25, 2021 by wachunga