So ONI new content might arrive next Tuesday or Thursday

[Aeglefire]

I would love love love to see performance improvements, like reduced calculations or reduced procedural calls, anything that helps with performance will be tremendously helpful. Even a reduced graphics option. Love you Klei!

[SakuraKoi]

4 hours ago, Aeglefire said:

Even a reduced graphics option

Utterly meaningless. bottleneck is by far, by very far, CPU. Even if also allocated to GPU, the point is that ONI has lots of numbers to process and not many graphics. There is nothing like view distance, animations are little and not filthy foliage. Never mind water and shadow shenanigans. Heck, my cheap AMD R9 390 is like 12 times better in overall performance than required... and I was getting problems after a few hundred cycles.

In any event, I hope it is today (in 11h) since I need good news today, for I also might get some bad news today as well (very likely). Of course two good news would be very sweet indeed. 

[Lancar]

12 minutes ago, SakuraKoi said:

Utterly meaningless. bottleneck is by far, by very far, CPU. Even if also allocated to GPU, the point is that ONI has lots of numbers to process and not many graphics. There is nothing like view distance, animations are little and not filthy foliage. Never mind water and shadow shenanigans. Heck, my cheap AMD R9 390 is like 12 times better in overall performance than required... and I was getting problems after a few hundred cycles.

How about a heat transfer calculation scaling option? ie, the ability to set the amount of heat transfer calculations for same-to-same materials per tick in the options menu. selections could be: 1, 1/2, 1/4, 1/8. It wouldn't affect us much if the temperature transfer of oxy to oxy would only happen 1/4 as much, it'd just update in bigger chunks, but if we got the machine to handle it we can scale it up.

[storm6436]

Given most of the back end physics sim is essentially a giant hateful ball of matrix operations, it'd be interesting to see if they could offload some of the calcs to gfx cards that have the appropriate hardware capability and excess capacity. 

[Gurgel]

2 hours ago, storm6436 said:

Given most of the back end physics sim is essentially a giant hateful ball of matrix operations, it'd be interesting to see if they could offload some of the calcs to gfx cards that have the appropriate hardware capability and excess capacity. 

Given that it is likely a very sparsely populated matrix (and likely not even implemented as matrix), I very much doubt that. 

2 hours ago, SakuraKoi said:

Utterly meaningless. bottleneck is by far, by very far, CPU. Even if also allocated to GPU, the point is that ONI has lots of numbers to process and not many graphics. There is nothing like view distance, animations are little and not filthy foliage. Never mind water and shadow shenanigans. Heck, my cheap AMD R9 390 is like 12 times better in overall performance than required... and I was getting problems after a few hundred cycles.

Indeed. Do Alt-S, zoom out to see everything and things just continue to move. 

[storm6436]

Well, given the topology is tile based with multiple layers, a matrix seems the native form.  Sure, data wise it would probably be either multiple 2D arrays or one 3D array, but arrays are topologically matrices (logic-wise)... not sure how else one would code the layout in an efficient manner.  I mean, if you were a masochist you could go for a linked list with horizonal and vertical links... but that's just an array with extra steps. 

[Gurgel]

3 hours ago, storm6436 said:

Well, given the topology is tile based with multiple layers, a matrix seems the native form.  Sure, data wise it would probably be either multiple 2D arrays or one 3D array, but arrays are topologically matrices (logic-wise)... not sure how else one would code the layout in an efficient manner.  I mean, if you were a masochist you could go for a linked list with horizonal and vertical links... but that's just an array with extra steps. 

There are quite a few options. For example, there could be several queues of different priorities, based on expected speed of change and other things. The high priority ones are just processed more often than the low priority ones. Also, you can have simple "add this every cycle" done more often and more rare "recalculate what to do every cycle". No matrix involved. The only thing you need is the neighboring tiles and that is simply (x-1,y),(x, y-1),(x+1,y)(x,y+1). The tiles itself can simply be stored in a 2-dim array. All pretty standard, nothing that says "matrix" in there. 

Note that "matrix" has special meaning to CS folks, a 2-dim array is not a "matrix" in most cases. Typically you only talk about a "matrix" when you want to do matrix multiplication or eigenvalue calculation or something like that. Just having a 2D coordinate system is a 2D coordinate system, not a matrix.

[ToiDiaeRaRIsuOy]

As someone totally not familiar with whatever you guys means with matrices, outside mathematics, 2D/3D Arrays adn topology matrices, I feel I can only do one thing to contribute to that discussion:

 

[old_fantom]

On 6/22/2019 at 7:24 AM, camelot said:

What things excite you guys the most?

 

An intelligent woman who could love me in spite of my mental illness.  If you find one, let me know...

[storm6436]

12 hours ago, Gurgel said:

There are quite a few options. For example, there could be several queues of different priorities, based on expected speed of change and other things. The high priority ones are just processed more often than the low priority ones. Also, you can have simple "add this every cycle" done more often and more rare "recalculate what to do every cycle". No matrix involved. The only thing you need is the neighboring tiles and that is simply (x-1,y),(x, y-1),(x+1,y)(x,y+1). The tiles itself can simply be stored in a 2-dim array. All pretty standard, nothing that says "matrix" in there. 

Note that "matrix" has special meaning to CS folks, a 2-dim array is not a "matrix" in most cases. Typically you only talk about a "matrix" when you want to do matrix multiplication or eigenvalue calculation or something like that. Just having a 2D coordinate system is a 2D coordinate system, not a matrix.

Not trying to be argumentative in the first paragraph, just laying out my background so neither of us waste time assuming where the other guy's coming from.    I'm well aware of the CS folks' of a matrix, having had the benefit of being a CS major my first trip through college.  Real life interfered, so after a segue involving military service and a career in IT as an admin, engineer, and programmer of things, this time around I'm a semester or three away from a physics degree.  Actually dropped the math dual major last semester due to financial constraints, might pick it back up once I have the money.

 Given it concerns itself with only the programming implementation, the problem with the CS meaning you gave is that it's one part limited and basically one part wrong-- one being that it describes only the programming implementation and it presumes only a 2D array.  Lastly it's wrong because 2D/3D arrays are still matrices from a data visualization standpoint.  So are structs, depending on how they're built.  An MxN array is topologically identical to an MxN matrix.  Similarly a 3D array (MxNxO) is the same as a MxN matrix row augmented O number of times (one of several ways to implement it, I'll point out.)   Really, CS is just math with a little more bookkeeping..

 Where that gets to be relevant is that when I referred to the game being a massive snarl of matrix operations I was referring specifically to the map data and the multiple layers that get stored.  The map data would certainly not be a sparsely populated matrix, array, or struct.   nor would it make much sense to store map data as a queue. And when I was referring to hardware acceleration, I was more pointing out that there's an interesting overlap in possibilities -- Were the map be laid out like an array/struct/whatever, provided it's topologically similar to a mathematical matrix, then every operation affecting every map tile in the physics simulation is a small piece of a much larger, much uglier simultaneous equation... given that simultaneous equations are linear algebra and both AMD and NVidia have compute libraries for accelerating linear algebra... the solution seemed to suggest itself.

 But... your suggestion of essentially a "Compute tile X, then X+1, etc" row/column (or even neighbor) scanning implementation is going to get you sooner or later because you'll end up with weird eddies, ripples, and counter-factual variations that are entirely the result of scan order you've implemented.  I got that point amongst many others firmly beaten into my head in my numerical analysis class and a few other upper tier math courses.  Turning the whole grid into a matrix with properly defined row/column/neighbor operators means you can crunch the entire map in one pass without the lovely artifacts that come out of progressive scan methods.

 Problem is, doing such a thing isn't necessarily easy and isn't necessarily something just any programmer is going to be able to do. And depending on the complexity of the array, its operators, and the amount of iron (ie. memory/IPC) you have to throw at the problem, there's a decent chance that without heavy optimization it's not going to be a whole lot faster, if at all.   Though, my point revolved around hardware acceleration and since we both know ONI doesn't really stress the vid card it seems feasible that perhaps chucking some what should be just matrix ops at hardware designed to be ridiculously good at matrix ops might not be a bad idea. 

[Gurgel]

Ah yes. But you see, a "matrix operation" is where the terminology issue becomes even clearer. "Matrix operations" are completely different from "map operations". The first one are things like addition and multiplication. The second are things like looking at neighbors because of interactions, looking at distances in Manhattan-metric, path-finding and the like.

Incidentally, you do not get those "ripples". The numerical analysis folks are a different community that does not really understand what is being done here. ONI is more like a finite element simulation and that is completely different. In particular, it has a locality property that you do usually not get in matrix operations and that is one more thing why "matrix" is the wrong mental model here. And, of course, you put those calculation results into a new 2D map, neatly eliminating all those "scan order" effects completely. Your numeric analysis folks were probably concerned with cases where that is not possible or represents a massive performance impact.

But the real issue is that these calculations are simple and memory I/O is the limiter, not CPU power. The code will be completely in 1st level cache and loading from RAM will be the thing that determines speed. For that situation, graphics card use slows things down. Graphics cards are mostly an accelerator for complex calculations that are not data- or I/O-sensitive.

I also do not agree on your non-sparse argument. Sure, every tile has to go somewhere. But if you separate this into tiles that need fast updates and ones that are fine with slow ones, you can expect massive performance gains. And then things get to be sparse, because you most definitely want to avoid having to pull the whole map from slow main memory every calculation cycle. An example on how to separate could be that you put all tiles into the "slow" class that are still surrounded by original tiles. And for an even slower class, do all tiles that have not yet been discovered. A fast set could be all tiles that are artificial (build by dupes) and an even faster set could be all tiles that have been build by dupes in the last 10 cycles. Or course, this comes with having tiles in more than one of these subsets.

I am not a fan of comparing credentials, but just to put things into perspective, I have CS PhD from one of the best tech universities on the planet. I do academic teaching, consulting, engineering and engineering failure analysis and paper reviews regularly. Don't let that stop you, I am very much trying to rely on argument by merit. Except for use on people that are clueless, I think an argument from authority is not worth much. I only use this when people are so clueless they have no clue they are clueless _and_ the cause is important. 

[OxCD]

Me also I have some knowledges : I know how to do a Rubik's cube 3x3 & 4x4.

Probably no one cares, but I'll be glad to teach it.

However, I still don't know where it could help into ONI... solving the artefact one maybe ?

...

No sarcasm guys just free trollin' (a bit), trying to provide to potential readers a life belt into this ocean of technical knowledges !

Kinda like this commu, you need to enjoy using your brain, and it feels that way many time

[Nightinggale]

It's a fine and interesting debate, but I wonder how useful it is. I mean nobody looks in the code to figure out what we have and then say something about how to use or modify the existing data structure to improve performance.You make me think of Stephen Hawking's talk about time travel. He came up with a way to travel forward in time, but admitted it wouldn't have any practical applications because you would need a spaceship and orbit a super massive black hole.

If we are to look at the existing code, the most interesting part is the Grid class. It has a bunch of static pointers, which are used as arrays. This means index access to temperature, mass, element, insulation, germ count etc for all the cells. The index goes left to right, top to bottom, meaning all the cells are in fact viewed as long 1D arrays.

One thing I notice is that some of the arrays are of bytes. This means they aren't thread safe. I'm not saying it's wrong. It's just an observation. Using just a byte when a byte has enough memory means using less memory, hence being able to fit more cell data into the CPU cache.

7 hours ago, Gurgel said:

But the real issue is that these calculations are simple and memory I/O is the limiter, not CPU power. The code will be completely in 1st level cache and loading from RAM will be the thing that determines speed.

I fully agree with this statement. It makes the most sense from both reading the code and the benchmark thread results. This means a major factor in performance is how efficient the CPU cache is used. I wonder if it would make sense to make a Cell class, which stores the contents of those arrays locally. If we look up temperature for a cell, we likely need mass and insulation for that cell too. It would make sense to keep those next to each other in the memory to fetch all into the cache at once.

You could say we have a different version of the debate of linked list vs vector discussion. The underlying mechanic of CPU cache utilization is at work here too.

7 hours ago, Gurgel said:

I am not a fan of comparing credentials, but just to put things into perspective, I have CS PhD from one of the best tech universities on the planet.

Sadly PhD seems to be the most common education level of internet trolls. This makes them so great experts that we should all take their word for whatever they say without arguments or documentation. This is then used to tell mostly uneducated people that "I'm the smart guy I always wanted to be in school" or that the Earth is flat or something else completely fictional. That is just one of multiple reasons why hiding behind a title such as PhD isn't the brightest approach online... or offline for that matter. I have seen people who actually have a PhD who seems downright crazy.

Having said that, you stand out as more real because you actually take your time and actually go into the topic and reply to replies in a meaningful way. Cross checking your statements with prior knowledge also makes sense and I say that with my master degree in engineering from "one of the best tech universities on the planet". As I just mentioned, adding an undocumented title makes all the difference

I don't have a beef with you or your credentials. I will just point out that my experience is that significantly more than 50% of titles mentioned online from people online seems to be fake. I'm not counting verified accounts from university professors or something like that where it's documented. I'm talking about regular forum accounts and similar not officially linked to a real person. Because of this, mentioning a title, particularly PhD sets off "increased troll risk" alert. It's just a friendly reminder of the sad reality of living "a World Where Facts Don't Matter".

[Gurgel]

2 hours ago, Nightinggale said:

Sadly PhD seems to be the most common education level of internet trolls. This makes them so great experts that we should all take their word for whatever they say without arguments or documentation. This is then used to tell mostly uneducated people that "I'm the smart guy I always wanted to be in school" or that the Earth is flat or something else completely fictional. That is just one of multiple reasons why hiding behind a title such as PhD isn't the brightest approach online... or offline for that matter. I have seen people who actually have a PhD who seems downright crazy.

Having said that, you stand out as more real because you actually take your time and actually go into the topic and reply to replies in a meaningful way. Cross checking your statements with prior knowledge also makes sense and I say that with my master degree in engineering from "one of the best tech universities on the planet". As I just mentioned, adding an undocumented title makes all the difference

I don't have a beef with you or your credentials. I will just point out that my experience is that significantly more than 50% of titles mentioned online from people online seems to be fake. I'm not counting verified accounts from university professors or something like that where it's documented. I'm talking about regular forum accounts and similar not officially linked to a real person. Because of this, mentioning a title, particularly PhD sets off "increased troll risk" alert. It's just a friendly reminder of the sad reality of living "a World Where Facts Don't Matter".

That is basically one reason why I do not like to give my credentials. I cannot prove them without doxing myself, and at the same time it is not really possible to go much higher. That makes using them a pretty unfair thing. They also do not add to the discussion. The discussion is about facts, not about credentials. But if I give them, I will give them in full. I will not reduce them to improve credibility. Getting a real engineering PhD was a lot of work and, at times, pretty painful.

Well, funny story, one guy a long time ago actually did find out who I was and then accused me of being the janitor breaking into my office to post things. That one still makes me smile. It does show nicely that people do not even believe you have that title and that publication list and the expertise that comes with them when the facts stare them in the face. Hence credentials are pretty useless as a means to "convince" people anyways. They will just stick to their convictions, no matter what.

I can also attest that there are numerous people that claim a PhD in the real world while not actually having one. (We look for the PhDs of customers at work when anyone claims one, so we can see what their expertise is and how best to talk to them. Funny thing is that we do not find about half of the claimed ones at all. And there is no way so many people are doing secret ones....) And I can attest to the fact that there are a lot of people with genuine PhDs that have an extremely narrow expertise and basically have no clue outside of that. 

[beowulf2010]

20 minutes ago, Gurgel said:

And I can attest to the fact that there are a lot of people with genuine PhDs that have an extremely narrow expertise and basically have no clue outside of that. 

Funny enough, these were some of my favorite professors back in my college days. Absolutely brilliant in their field (music history, specifically Charles Ives, specifically the Concord Sonata), apologizing to the trash can after they run into it, and doesn't know how to use their email (this was the late 90's). Loved that class... 

[Nightinggale]

21 minutes ago, Gurgel said:

Well, funny story, one guy a long time ago actually did find out who I was and then accused me of being the janitor breaking into my office to post things.

You should be proud. Janitors are excellent people. Take for instance the janitor at the university of Edinburgh. Part of his job was to clean blackboards to make the ready for the morning lectures the next day. He had a habit of reading what was written before erasing everything and one day he read about Earth's elliptical orbit about the Sun and how it is more or less elliptical over time. Here he had a lightbulb moment because he thought of the blackboard from another lecture, which was about ice ages and hot periods between those. If making the orbit more elliptical increases the average distance to the Sun, wouldn't the shape of the orbit control the temperature, hence when ice ages start and end?

If a janitor can make a bigger scientific breakthrough than a number of those people with PhD today, then obviously being mistaken for a janitor will have to be a good thing, right?