Full map (all worlds) zoomed out + map science

[psusi]

17 hours ago, Gurgel said:

Not really surprising, though it would have been nice if they were separate for performance reasons.

Yea, I thought for sure they were separate since that way they could each run in their own thread.

15 hours ago, AntiBlueQuirk said:

But from a development standpoint, separating the single grid into virtual worlds is way, way easier than making a multi-grid system. Because basically every part of the codebase assumes that there's only one grid right now. Changing all that code to be multi-grid aware would be a huge refactor, so I can see why they didn't do it. (I'm sure implementing the current system was already a lot of work.)

No; it's as easy as falling off a turnip truck.  Since they seem to be using C++, I'm sure that the current code doesn't just use a single global variable for the one and only instance of the map, but even if it did, all you have to do is change the definition of that variable to be thread local and poof, each thread has its own world.

[pether]

33 minutes ago, psusi said:

No; it's as easy as falling off a turnip truck.  Since they seem to be using C++, I'm sure that the current code doesn't just use a single global variable for the one and only instance of the map, but even if it did, all you have to do is change the definition of that variable to be thread local and poof, each thread has its own world.

The only thing in threads "as easy as falling off a turnip truck" is introducing new bugs that are really hard to debug. If something works, I see no point to smash threads into it. 

Also - I don't understand how things would change for the performance. If you have to calculate 40.000 cells, there is no difference if you split them in 4 chunks 10.000 each - you still need to calculate your maths 40.000 times. "But threads!" you say... Why can't we divide those calculations into threads while we are on a single map...? Why you think that dividing map into several parts is required to introduce threads?

[AntiBlueQuirk]

1 hour ago, psusi said:

...all you have to do is change the definition of that variable to be thread local and poof, each thread has its own world.

When I was talking about single-grid vs multi-grid, I was still assuming a single-thread scenario. Without considering multithreading, I speculate that single-grid solution in the game was still *way* easier to implement. Even in single-threaded code, splitting the tile-grid into multiple separate grids would be a *lot* of work.

But a multi-grid world isn't even necessary to implement multithreading. If you treat each world as unable to interact with tiles outside of it, then it should be possible to update each region of the grid on a separate thread. I think. I'm not Klei, so I can only speculate. Multithreading has setup and synchronization costs. The only way to tell if it's worth it is to implement it, and that's less time for Klei to work on other features. (Like adorable Plug Slugs.)

Thread-locals wouldn't be useful for things like critters because critters aren't attached to threads. A critter needs to know what world it's in, regardless of what thread is working on it. Right now it knows that based on position alone.

[psusi]

2 hours ago, pether said:

The only thing in threads "as easy as falling off a turnip truck" is introducing new bugs that are really hard to debug. If something works, I see no point to smash threads into it. 

Because we now want to have multiple maps instead of just one, so it naturally follows that each map should be its own thread.  They are supposed to be independent from each other and that's exactly what threads are.

2 hours ago, pether said:

Also - I don't understand how things would change for the performance. If you have to calculate 40.000 cells, there is no difference if you split them in 4 chunks 10.000 each - you still need to calculate your maths 40.000 times.

If you have 4 people each working on their own 10,000 cell map, they can get it done in 1/4th the time of having one person do one big 40,000 cell map.  Duh.

2 hours ago, pether said:

Why can't we divide those calculations into threads while we are on a single map...? Why you think that dividing map into several parts is required to introduce threads?

Because it either isn't possible to divide one map into multiple threads, or it is infeasibly hard.  Running each independent map in its own thread is stone cold simple.

1 hour ago, AntiBlueQuirk said:

Multithreading has setup and synchronization costs.

Not in this case it wouldn't.  You only need to synchronize when they need to interact, which means only when a rocket or warp pipe sends things from one map to the other.  Otherwise the worlds are totally independent and can just run freely in their own thread and not know or care about the others.  All of that time spent doing all of the temperature calculations for each world can proceed in parallel without any concern.  Likewise the pathfinding can easily be done in parallel since there should be no path from one world to the other, each thread only needs to consider its own world.

[AntiBlueQuirk]

2 minutes ago, psusi said:

Not in this case it wouldn't.  You only need to synchronize when they need to interact, which means only when a rocket or warp pipe sends things from one map to the other.

You still have to synchronize the simulation with the rest of the game. The rest of the game logic can't proceed until all needed tiles have been simulated, multi-threaded or not. Likewise, pathfinding is more difficult to multi-thread, because dupes make decisions based on pathfinding, as well as pathfind based on game state. So they depend on each other. There would have to be lots of synchronization there.

I agree that there's potential there, but again, it's something only Klei knows the implementation cost of, and the performance gains might not even be worth the cost. It *could* be that the major bottleneck for the simulation is memory bandwidth, and having multiple threads won't help that much. This is discussed a few times in Factorio's Fridays Facts, where they parallelized something, but hit a memory bottleneck instead.

1. Simulating tile physics in parallel for different worlds is probably possible.
2. It may or may not actually improve performance.
3. It would have an implementation cost (developer time).

I'm all for optimizing the game, but sometimes optimizations don't pan out the way one predicts. I'm sure they are actually working on optimizations. Some of the patch notes mention it, and the fact the game limits simulation to discovered worlds is evidence of it. But the potential gain of multithreading may not be worth the development cost when they have other stuff to work on. (And optimize!)

[psusi]

32 minutes ago, AntiBlueQuirk said:

You still have to synchronize the simulation with the rest of the game. The rest of the game logic can't proceed until all needed tiles have been simulated, multi-threaded or not. Likewise, pathfinding is more difficult to multi-thread, because dupes make decisions based on pathfinding, as well as pathfind based on game state. So they depend on each other. There would have to be lots of synchronization there.

All of that only depends on one world and does not care about the others, so there is no need to synchronize between worlds.

34 minutes ago, AntiBlueQuirk said:

It would have an implementation cost (developer time).

Yes, but very, very little given how naturally separate worlds lends itself to multithreading.  You can't ask for a more perfect problem to try to multithread.

 

[AntiBlueQuirk]

I've said my piece. I'd rather talk about the cool science to be done with the new maps.

[qda]

All this synchronization discussion made me thought about something  :

What if, the planetoïds actually moved away from each other, fast enough that they would actually be, relatively, desynchronized.

For the gamer, building a tile would take the same time (real time) on any world. However, while focused on a specific world, tasks on another would be completed faster (slightly ?).

Klei, please hire Einstein.

[nessumo]

Great Scott, no! Imagine rocket going from asteroid A to B and back before it arrives to B just because you were constantly focused on B. Just because there are no dups grandparents to be killed doesn't mean there are no paradoxes!

[qda]

More paradoxes = more fun !

Forgetting the relativity stuff, this would be an incentive to invest in better rocketry in a decent time. If you think about it, after having a stable colony, which is quickly and easily achieved once you played a bit, literally nothing is a pressing matter anymore, single or many worlds. Now, if planetoïds moved away 1 tile from the starting world every 200th cycles (setting adjustable by the player), that would make space "race" an actual thing. And with ultra-late tech + nuclear thingies + some knowledge or artifact hidden deep within a far and hugely hostile asteroïd, you could counter the seemingly unavoidable and sad fate of the total isolation of each world : either being able to build warp stuff, or making space flight times relative too so they wouldn't change, or even physically stopping planetoïds from moving away from each other.

That would be an epic end-game objective.

[Carcer_]

I wouldn't trust dupes with relativity. They would 100% find a way to potentially kill themselves before they are printed.

[JahwsUF]

@psusi's right on this.  In terms of concurrency, each asteroid is "embarrassingly parallel."  Outside of rockets and warp pipes, there is literally zero interaction between maps.  Unless things were implemented with a lot of globals, it should be very easy to, essentially, run multiple copies of the base game's engine (one per thread) without the need for much refactoring.  Well, before the new features are added, anyway.

There's absolutely no question that in a multi-base late-game, this approach would easily yield near-linear speedup until you run out of either independent bases or physical cores - with no extra optimization.  Why introduce new issues within a single map when you can keep the same old code and just run the proven thing in multiple instances?

15 hours ago, AntiBlueQuirk said:

This is discussed a few times in Factorio's Fridays Facts, where they parallelized something, but hit a memory bottleneck instead.

Factorio doesn't have multiple, completely-separate maps.  The in-game world is unified, even if you can represent it as a set of neighboring blocks.  It's an entirely different and unrelated problem.  Even if it seems similar, the fact is that neighboring blocks there interact with each other, unlike individual ONI asteroids.

In fact... by splitting things into multiple, smaller maps, it'd help ONI to avoid memory bottlenecks and to better exploit cache locality.  There's no reason for asteroid 1's thread to even load asteroid 2's map, and each processor core manages its own, separate cache.

------

All of that said...

15 hours ago, AntiBlueQuirk said:

But the potential gain of multithreading may not be worth the development cost when they have other stuff to work on. (And optimize!)

Depending on their current priorities, this is always important to consider.  UI elements, like the notification system, would definitely be trickier - some of their data would be managed by other threads, after all.  Nothing too crazy, though.

I'd be very surprised if they don't have plans to go multithreaded with this.  Of course, it requires significant development time and cost.  I imagine what we currently have is a 'simpler' demo build for demo use, while a multithreaded engine is in the works in the background.  Multithreaded stuff is always trickier to get right, so I can easily understand wanting to more thoroughly vet it on a separate side branch while presenting more obvious features to us publicly right now - those "obvious new features" generate more user-testing interest, after all.

[NullDragon]

4 hours ago, JahwsUF said:

Factorio doesn't have multiple, completely-separate maps.  The in-game world is unified, even if you can represent it as a set of neighboring blocks.  It's an entirely different and unrelated problem.  Even if it seems similar, the fact is that neighboring blocks there interact with each other, unlike individual ONI asteroids.

Factorio actually does have LuaSurface.  In normal gameplay there is only ever one surface, "nauvis", representing the main Factorio planet. Mods can use surfaces to create tardis-like buildings or caves with interior surfaces, or even entire planets that can be traveled to. You are correct that in normal vanilla gameplay multiple surfaces are generally not used(AFAIK), however.

[AntiBlueQuirk]

Today I learned that in the Sandbox mode, there's a new tool: the Radiation tool!

Overall, it's not that interesting, since it seems like radiation isn't simulated yet. However, the tool lets you access the otherwise inaccessible radiation overlay, and I learned something neat:

Spoiler

the Crushed Satellite is radioactive!

Edit: also, if you erase the radiation using the tool, it appears that the satellite is actually emitting radiation! It's not a map-gen thing. I suspect the satellite is adding radiation, but since radiation isn't enabled in the alpha, it's not dissipating. It gets projected out in little rays like this:

Also, there's a *lot* of radiation around the crash itself. I think it's been building up there over the whole time I've been playing!

I haven't found anything else radioactive. I suspect anything that would be radioactive has the emission disabled for the alpha. I think the satellite was an oversight.

 

[test447]

The parts of Oxygen Not Included that we can know are multithreaded (from dev posts and looking at the code) is the pathfinding. According to the game's code it uses a thread pool with a number of threads equal to your computer's logical cores, and pathfinding tasks are sent out to this threadpool to compute. So we know that at least the game's pathfinding system can take advantage of newer CPUs with many threads.

Additionally we do not know if the game's backing sim is multithreaded (the backing sim is the part of the game that computes things like heat transfer, gas movement, etc...) since the sim is written in (presumably) c++ and is compiled natively to each platform that's targeted (so way too annoying/difficult to reverse-engineer). I would have to guess that at least in the case of the backing sim, the separate asteroids are as "embarrassingly" simple to parallelize so perhaps the devs have already done this or its in the works.

But the rest of the game fortunately/unfortunately runs in the Unity engine which (as far as I'm aware) cannot really be easily parallelized. So even if in theory it's much faster for the game to have been running with each asteroid handled by a separate instance of the game engine that only processed that asteroid's info and such, the game wasn't built up that way originally and probably doesn't lend itself well to that.

I guess all I'm saying is to temper expectations and not expect multithreading in much more places than pathfinding and the sim since the other locations that multithreading could be introduced probably aren't worth the dev time. I expect we'll see pretty big performance improvements just from the smaller asteroids making pathfinding cheaper alone.

[misotoma]

On 11/29/2020 at 4:17 PM, 2tallyGr8 said:

If so, by zooming out you could place pipes that went directly from one asteroid to another.

I wonder if you can, avoiding neutronium walls obviously, just pipe resources from point A to B to C lol would be a nice bug we could abuse

[Primalflower]

when i initially found out about how DLC maps work, I was pretty much instantly that "how was gym" picture...but something about how the game has capability to allocate space for more rocket cockpits makes me feel much, much better about it...

[babba]

Great thread In a perfect world I could copy and paste parts of my "old map" from the "old game" in to the new dlc map zones...

Do you know or does anybody know if its possible to copy content from one map to the new dlc format map ? @AntiBlueQuirk

So adding really new things to the game ( like a space station as example ), which could require some new designed additional space, would be adding more map zones to the new dlc file format/size or using your described Factorio mod way @NullDragon

On 12/1/2020 at 4:01 PM, NullDragon said:

Factorio actually does have LuaSurface.  In normal gameplay there is only ever one surface, "nauvis", representing the main Factorio planet. Mods can use surfaces to create tardis-like buildings or caves with interior surfaces, or even entire planets that can be traveled to. You are correct that in normal vanilla gameplay multiple surfaces are generally not used(AFAIK), however.

Feedback would be great

 

[The Plum Gate]

Having critters path out into space is probably bad - until critters from space can path in.

Which is as good an excuse as any for invasive species or swarms of space bugs.

[babba]

13 minutes ago, The Plum Gate said:

Having critters path out into space is probably bad - until critters from space can path in.

Which is as good an excuse as any for invasive species or swarms of space bugs.

This is how Space Invaders was born

[natanstarke]

On 11/30/2020 at 2:20 AM, AntiBlueQuirk said:

You can't edit the borders of the map, even in sandbox mode. Most tools just don't extend outside the build area. The fill tool does, but it's much hard to control since there's so much neutronium, and it's probably a bug. So if you remove the neutronium between adjacent worlds, you can get gasses and liquids to flow between them. But the border tiles count as space tiles, so any gasses or liquids are going to quickly evaporate. And you can't build on the borders in any case, so there's no good way to control it. If you make the border tiles solid non-neutronium, they can transfer heat.

As far I can tell, the map is still one big simulation. One optimization is that the game only simulates tile physics inside the bounds of discovered worlds, but that only helps up until you've discovered everything. The world borders don't seem to affect the simulation at all, except that they're made of neutronium. The pathfinder definitely doesn't seem to care about world boundaries, but this is only relevant for flying things. Nothing else could ever path over a world border anyway. (I expect flying creatures will probably get a world border check in an upcoming patch, since it also affects jet-suits.)

The parallel simulation thing is interesting. Right now worlds can interact with each other through tile physics, but there's no reason they need to. So simulating each world in parallel could work in theory. But since tile simulation happens in native code, only Klei could tell how practical it is to do that, or even if it's worth it. As others have noted, a lot of late game slowdown comes from all the stuff in the world, not the number of tiles. Pathfinding could be parallelized, even with a single world, since critters can't affect each other's paths. But dupes decide what to do based on pathfinding, and that can affect other dupes, so that could be a problem. Even then, sometimes multithreading just doesn't actually help that much. See Factorio Friday Facts #215. It's one of those things only Klei can really do, and it might not be worth the development cost.

Deciding which parts of the map get simulated happens in managed code however, so people could write mods to change when and where "off-world" simulation happens. Right now, once you've "discovered" a world, it starts getting simulated, and there's no way to "undiscover" a world. A mod however, could let you freeze worlds without any dupes in them. That's pretty exploitable, but might be necessary when people decide "I want to play with 20 asteroids!" Which is definitely going to happen, I'm sure.

O want to play with 20 asteroides!!!! 

[RageLeague]

On 11/30/2020 at 7:47 AM, psusi said:

Yea, I thought for sure they were separate since that way they could each run in their own thread.

No; it's as easy as falling off a turnip truck.  Since they seem to be using C++, I'm sure that the current code doesn't just use a single global variable for the one and only instance of the map, but even if it did, all you have to do is change the definition of that variable to be thread local and poof, each thread has its own world.

The game runs in Unity, so it uses C#. Also while C# have no global variables, it does have static variable for classes, and it is useful for getting the singleton instance of an object, like the map grid of the base game. Changing it makes functions that rely on the fact that certain objects are singleton not work anymore, and they probably need to rework a lot of the code. Not sure how hard that would be.

Javascript don't count.

[AntiBlueQuirk]

On 12/17/2020 at 8:07 PM, The Plum Gate said:

Having critters path out into space is probably bad - until critters from space can path in.

AFAIK, that's fixed now. But I haven't played since the release, so I haven't tested it.

 

On 12/14/2020 at 1:55 PM, babba said:

Do you know or does anybody know if its possible to copy content from one map to the new dlc format map ? @AntiBlueQuirk

So adding really new things to the game ( like a space station as example ), which could require some new designed additional space, would be adding more map zones to the new dlc file format/size or using your described Factorio mod way @NullDragon

There may be some difference between the new and old map formats, but I don't think it's anything major. It's just a difference in content. The only difference I know of is that the DLC map has some extra data to separate the map into submaps. It was possible at one point to load base game maps into the DLC, but I think there were compatibility problems. (And you didn't get any other asteroids anyway.) Sounds like at some point they plan to merge the DLC into the main game, so there's that.

I don't think there's any sandbox tools for editing the "system map", and normally asteroids are only created at world gen, but someone could probably fix that with a mod, including "importing" asteroids. So "Possible now?" Not really. "Possible ever?" Probably. (Not volunteering, btw.)

Yeah, they already add new submaps at runtime. That's where the rockets get their space. It would be really easy for a mod to allocate their own new submaps. Basically you'd just have to make a space-station POI template, then add some "space station" module you could deploy on the system map. You could even make it expandable if you wanted by placing/removing inter-map doors and pipe connections during play.

 

On 12/18/2020 at 5:18 PM, natanstarke said:

O want to play with 20 asteroides!!!! 

Called it.