Ipsquiggle

Understanding Shards and Migration Portals

64 posts in this topic

Migration involves using in-game portals to travel between actual game servers. Currently, the Cave Entrance/Exit is an example of a migration portal. Effectively, each level of the world is run on a different server, and the portals allow players to travel between them.

Although we provide a default configuration where there is an overworld and a cave, you can actually arbitrarily link a large number of servers together, and arbitrarily generate the worlds on each one to create your own unique topologies (for example, 3 overworlds connected to each other by a single shared cave!).

We'll discuss this in two parts. The first part is setting up the servers to communicate with each other, and the second part will be the configuration of the portals, which actually create the links.

 

Part 1: Connecting your Servers

There are two possibilities: either your servers are running on the same machine (easier) or on separate machines (more exandable/less lag). The setup is basically the same for both, except that on the same machine, all servers will point to a local address (127.0.0.1) and share the port space, but on separate machines shards must be aware of the master server's address, but each can configure ports as they please.

Before diving further, please http://forums.kleientertainment.com/topic/64552-dedicated-server-settings-guide/]familiarize yourself with the .ini files.

 

cluster.ini

Remember: Every server must have the same cluster.ini. If you are running on the same machine out of the same config directory, this happens automatically, but if you are on separate machines, you'll have to copy any changes to this file to all machines.

So, the things you'll need to pay attention to in cluster.ini are in the [shard] section:

shard_enabled: Set to true, obviously. :)
bind_ip: Default is fine for single-machine setups, 0.0.0.0 is necessary for multi-machine setups.
master_ip: 127.0.0.1 for single-machine setups, the master's IP address for multi-machine setups.
master_port: Must be set to the port of the master. This is discussed further in the settings.ini section below.
cluster_key: This is a simple privacy measure, the master will reject shards that don't share the key. It prevents other servers from nefariously connecting to your cluster.

You may wish to customize many of the options in the [network] section to your liking as well!

 

server.ini

Each server has it's own server.ini. The first thing is to customize the [network] and [steam] section:

[network]
server_port: On the same machine, this must be unique among the servers, but for multi-machine servers, can be whatever. Pay special attention to the value you give the master server, as it must match the value you put in the master_port in cluster.ini.

[steam]
authentication_port and master_server_port: If you are doing single-machine, just make sure these are unique from the other servers and don't interfere with any other ports.

And naturally, you'll have to configure the [shard] settings as well:
is_master: Set this true for the master, and false for all other shards.
name: Name your shards! Required, but also makes understanding the log files and folders much easier.
id: Whatever value you put here is what will be used when linking the migration portals, which is explained below.

So, once all that's done, launch all your servers and you should see lots of nice messages on the master about the shards connecting! Yay!

 

Part 2: Linking your worlds

(punchline: Nuke1 has made a mod which simplifies the portal configuration, so once you understand the basics, check out his great mod and see if it suits your needs!)

This part is a little trickier. Right now, the migration portals (cave_entrance and cave_exit) are set up to automatically link exactly two servers together with no configuration.

Effectively what you have to do is, for each migration portal in your world (and you can add or remove them just like any prefab, with c_spawn() and :Remove()), you need to disable the automatic linking, and then point each portal at the world it should migrate to.

So for example, lets say you want three worlds connected in a chain: 1<->2<->3.

1) Load in to the master server (id: 1), and teleport to the first portal (with c_gonext('cave_entrance') or 'cave_entrance_open' or 'cave_exit', whichever is appropriate).

2) Stand next to the portal and run c_find('cave_entrance').components.worldmigrator:SetDestinationWorld("2", true). This will link the portal to world 2, and make the change permanent (i.e. turn off the automatic linking). Do this for every portal in the world, and they will be linked to world 2.

3) Now, go through the portal. It should take you to shard 2!

4) Here, you'll follow the same steps, except that for half the portals, you'll link them back to shard 1, and half you'll link to shard 3. Now the middle world is set up!

5) Finally, travel through one of the portals to shard 3, and link all the portals back to shard 2.

(Remember, if you have an uneven number of links, such as 5 links from 2->3, but 10 links from 3->2, the "extra" portals will just take you to the Multiplayer Portal in that world, so you may with to add/remove portals so that the links are equal.)

 

And there you go! You have 3 worlds linked! This process expands easily to any number of worlds and connections. However, traveling all over to configure many worlds can be a pain, so I do recommend Nuke1's mod linked above!

4 people like this

Share this post


Link to post
Share on other sites

@Ipsquiggle, up to how many slave servers can connect to a master server? And is it possible to have an overworld that connects to a caves world and then in that caves world you find sinkholes that connect to another world? Or do all slaves need to have their portal in the master and not slave to slave? 

Edited by Iking
1 person likes this

Share this post


Link to post
Share on other sites

@Iking,  I'm not sure if we have any kind of hard limit, but I'm guessing due to network traffic with the master there is a practical limit...

 

Any portal can link to any world, so you can create any layout you like. If that's not the case please let me know because that's the intention!

Share this post


Link to post
Share on other sites

@Ipsquiggle, I tried making a 3-level world with 2 kinds of configurations.

 

Configuration 1: (all worlds have their sinkhole to the 1st/master)

Overworld(master) -> cave1&cave2

 

Configuration 2: (cave2 should not have a sinkhole to the overworld)

Overworld(master) -> cave1 -> cave2

 

Neither configuration worked. I changed the ports properly meaning each had it's own unique port and shard name and I just changed the master_port in caves2 to point to the port of which world I want it to connect to depending on the configuration. In configuration 1, if cave1 isn't turned on then cave2 will connect to the overworld.

 

It seems like the master can only accept a connection from just 1 world. And that the slave (cave1) cannot accept a connection from another slave, which does make sense since it should be a master to accept a connection.

 

I'm hoping there is a way to have multiple worlds (more that 2) connect to each other in either format. The ideal would be (1) to make a multiple level world like the adventure mode in DS (it would be nice if we could even use the map configuration of those adventure worlds also) but with sinkholes instead of the portal and (2) to be able to connect different servers like my friends multi-level server could connect to mine if both are online so we could co-op and help each other's server or lay siege to the other.

 

Is there another setup I could do in settings.ini to make both configurations above work? Since changing the ports doesn't seem to work.

 

Configuration 1 logs.zip

Configuration 2 logs.zip

Share this post


Link to post
Share on other sites

The only way to have multiple worlds connected is by being an admin, migrating to each one of them using

c_migrateto(worldId, portalId)

and then editing the worldmigrator component on each sinkhole/Friend-O-Matic.

for k,v in pairs(Ents) doif v.prefab=="cave_entrance" thenv.components.worldmigrator:SetDestinationWorld("8721381214", true)endend

That's the shard id.

 

You would need 10 sinkholes in the first world, 10 cave_exits on the second world, 10 extra spawned manually cave_entrances on the second world, 10 cave exits on third world. This means you only have to manually generate 10 cave_entrances on the second server.

 

For each hole generated, you would need to keep track of its id (which would be between 11-20).

local i=11for k,v in pairs(Ents) doif v.prefab=="cave_exit" thenv.components.worldmigrator:SetRecievedPortal("123814903289", i)i=i+1endend

Then in the third world, you go and set the cave exits as receiving portals from the second world using its shard id.

 

 

Basically, no, it's not a matter of changing the configuration.

The configuration is useful to determine which server is the master (shard id is 1) and the shard ids of the others.

1 person likes this

Share this post


Link to post
Share on other sites

@DarkXero, thanks for this. I will try this soon. However, will this fix the issue of the master being unable to accept connections from 2 slaves? I'm not a coder but it seems that these are based on the assumption that the 2 slaves are able to connect to the master and you just aren't able to migrate from one to the other automatically via sinkhole because of a lack of entrances/exits.

Share this post


Link to post
Share on other sites

When you create a shard network (this is, you have a master and N slaves), you can travel from one server to another one.

No matter if it's master or slave.

 

Example, you have a master and two slaves.

 

You can setup:

a) (from master to anywhere)

Master <-> Slave 1

Master <-> Slave 2

b) (chain, from master to slave 1, from slave 1 to slave 2)

Master <-> Slave 1

Slave 1 <-> Slave 2

c) (from anywhere to anywhere)

Master <-> Slave 1

Master <-> Slave 2

Slave 1 <-> Slave 2

 

The issue is in how the portals register and link with each other.

By default, there are 10 portals in master, and 10 portals in a slave.

They automatically link each other assuming there's one master and one slave, it's hardcoded.

Share this post


Link to post
Share on other sites

@DarkXero, I was finally able to connect the 2 slaves to the master. Apparently my mistake was that I didn't change the shardID of the 2nd slave when I copied the settings from the first, I only changed the ports originally. That's why the master was saying there was a duplicate slave.

 

Just to check the worldmigrator.lua file you're referring to is the one found in C:\Program Files (x86)\Steam\steamapps\common\Don't Starve Together Dedicated Server\data\scripts\components or is it another file? And do I just add the code at the very end? Sorry, I'm not really familiar with coding to be able to figure this out on my own and I couldn't find an online reference to help me.

 

Thanks so much for your help. :-)

 

Edited by Iking
1 person likes this

Share this post


Link to post
Share on other sites

You are not supposed to touch the code of worldmigrator.lua.

You are supposed to run code on the console.

 

1) Join a server

(We are in master now, shard_id is 1)

2) c_spawn("cave_entrance")

(Cave entrance will be id 11, automatically, since there were 10 cave entrances by default on generation)

3) Stand near the entrance, as c_find picks the closest thing

4) c_find("cave_entrance").components.worldmigrator:SetDestinationWorld(shard_id)

(This shard id will be the second slave)

5) Use the entrance

6) You will migrate to slave 2, from portal 11 from master

7) Since only 10 cave exits are here, you will go somewhere else, find a place to make a new exit

8) c_spawn("cave_exit")

(This cave exit will be id 11, automatically, since there were 10 cave exits by default on generation)

9) c_find("cave_exit").components.worldmigrator:SetRecievedPortal(shard_id, 11)

(This shard_id is 1 if it's master)

Share this post


Link to post
Share on other sites

@DarkXero, I was able to follow the instructions to make a portal from overworld to slave 2. It brought me to the portal at slave2 (it's also a caves server) but when I try to link the cave_exit of slave2 to the master I get brought to the portal of slave1. Can I use the existing cave_exits on slave2 to link with the master or do I need to spawn new/separate cave_exits?

Share this post


Link to post
Share on other sites
Can I use the existing cave_exits on slave2 to link with the master or do I need to spawn new/separate cave_exits?

 

You can do both, you just need to

c_find("cave_exit").components.worldmigrator:SetRecievedPortal(1, x)

with x being the portal id from where you came from master.

Share this post


Link to post
Share on other sites

Super cool that you're working through this, @Iking. (And thanks @DarkXero for all the assistance!)

 

I've modified the top post to explain more about the portals, I'm not sure why I left that out originally...

Share this post


Link to post
Share on other sites

@Ipsquiggle,@DarkXero, it took awhile but I was finally able to link the 3 worlds properly but I did it cascading that 10 default sinkholes from the overworld links to caves1 then 10 new/spawned sinkholes in caves1 links to caves2.

 

Many thanks to you both for helping and the explanations! I did this as an experiment because I'm going to try to make different worlds have different challenges to spice things up. :)

 

@Ipsquiggle, is there a chance that part of the DST roadmap will include the adventure mode from regular DS? Or at least the different map types for diversity?

 

Thanks again! :) 

Share this post


Link to post
Share on other sites

Hi, there is a small problem. Everything works, but the slave server work with normal biome, not like cave. Now I have got 2 normal world. How do I configure the slave server so that it works like a "true cave"?

 

thx: otosi

 

Share this post


Link to post
Share on other sites

Does anyone know what happens if you try to link worlds with different modes?  (survival and wilderness for example)

 

I kind of like the idea of having a "hidden" world in wilderness that can't be reached except by going through a survival surface and a survival caves, or maybe a cave inside a cave -- the interesting bit being having a permanent world linked to two temporary worlds.  Having it difficult to reach/find would help somewhat with griefing, I hope.

Share this post


Link to post
Share on other sites

Hi, there is a small problem. Everything works, but the slave server work with normal biome, not like cave. Now I have got 2 normal world. How do I configure the slave server so that it works like a "true cave"?

 

thx: otosi

you did add "worldgenoverride.lua" in your cave server settings, right? in the same folder as settings.ini

 

if you did, check your main server logs and see what it says when someone tries to enter a cave. Does it say anything about "Migration failed"? 

 

 

Edited by Shinray

Share this post


Link to post
Share on other sites

Hi, there is a small problem. Everything works, but the slave server work with normal biome, not like cave. Now I have got 2 normal world. How do I configure the slave server so that it works like a "true cave"?

 

thx: otosi

 

OK, i solved the problem. The slave config directory must be a worldgenoverride.lua file with this line:

 

return { override_enabled = true, preset="DST_CAVE"}

 

or a slave world will be a "forrest" world.

 

Thank you for the "DoNotStarveTogether_EasyConfigCaves" directory.

Share this post


Link to post
Share on other sites

How do I find the IDs of portals so that I can link them?

 

Are IDs unique to a specific shard, or unique across all shards?

 

Should I be able to use c_migrateto(...) to relocate to those cave entrances and exits?

 

If I'm making a world that requires a while to set up and configure is there a way to start a new game in that properly set-up world?  Alternatively, is there a way to pause time in a server while still being able to move around?

 

Thanks

 

Share this post


Link to post
Share on other sites

How do I find the IDs of portals so that I can link them?

 

Are IDs unique to a specific shard, or unique across all shards?

 

Should I be able to use c_migrateto(...) to relocate to those cave entrances and exits?

 

If I'm making a world that requires a while to set up and configure is there a way to start a new game in that properly set-up world?  Alternatively, is there a way to pause time in a server while still being able to move around?

 

Thanks

What I think I know so far:

So I figured out the way IDs work even though I don't have a good way of identifying the ID of a particular cave entrance or exit (IDs are unique to a shard, but not across all shards, they start from 1 and increment by 1). 

As far as I can tell c_migrateto(shard_id, portal_id) only works when moving to a different shard than the current one, and the only portal_id i'm aware of is that of the jury-rigged portal on any map (ID 1).  Also, c_migrateto(shard_id, 1) sometimes locates me at a mined surface cave entrance, even if it wasn't mined before (not sure if it's creating it or not).

 

Still nothing on pausing or changing dates backward.  Snapshotting system seems to incorporate state of cave entrance and exit, so rolling back appears to roll that back as well, though I'm not sure on this.

 

I also don't know if it's intended behavior but c_gonext("cave_entrance") only loops through unmined cave entrances.  If there's a way to loop over all cave_entrances and/or just mined cave entrances that would be very useful in trying to sort out my setup.  I also am not sure about this, but if c_gonext("cave_entrance") doesn't register the mined entrances, I suspect that also means that c_find("cave_entrance") doesn't find them either, thus dumpstering any commands built on that return result.

What I'm struggling with currently:

I am attempting to set up 3 cave maps and 3 surface maps.  The surface maps and cave maps should be paired as normal, and then the master (one of the surface maps) should have two additional cave entrances which should be paired, one each, with an exit in each of the two caves that aren't already paired with the master surface shard.

here's a crappy graph of what I'm talking about:

SurfaceSlave1      SurfaceMaster    SurfaceSlave2

    | | | | | | | | | |     /      | | | | | | | | | |    \    | | | | | | | | | | |

    CaveSlave1        CaveSlave2       CaveSlave3

Where each |, /, or, \ represents one bidirectional cave entrance/exit pair.

 

SurfaceMaster has shard ID:   1

SurfaceSlave1 ID:                    2

SurfaceSlave2 ID:                    3

CaveSlave1 ID:                        11

CaveSlave2 ID:                        10

CaveSlave3 ID:                        12

 

In my experience trying to construct this, two maps seem to be linked bijectively, but which two seems to depend on world generation(I've had it be a cave and a surface and often it's two caves).  All other entrances and exits link to the jury-rigged portal of one particular cave.

 

When attempting to follow DarkXero's instructions I load up a fresh game,

1. c_spawn("cave_entrance")

2. c_find("cave_entrance").components.worldmigrator:SetDestinationWorld(11, true)  where 11 is the shard id of CaveSlave1.

3. mine out the cave entrance and go through the sinkhole

4. Find myself in CaveSlave2, not CaveSlave1. Figure I'll continue anyway.

5. c_spawn("cave_exit")

6. c_find("cave_exit").components.worldmigrator:SetReceivedPortal(1,11)

7. I take the cave exit only to end up at the jury-rigged portal of CaveSlave2.  Assuming that maybe I need to set the destination world as well I do

8. c_migrateto(10,1)

9. c_find("cave_exit") until I reach the exit I had made (which should have ID 11)

10. c_find("cave_exit").components.worldmigrator:SetDestinationWorld(1, true) I assume this command is only ever needed for cave entrances but don't really know.

11. I take the portal only to end up at the jury-rigged portal of CaveSlave1.

 

In other words, I can't get transportation working in even one of the directions.  Is SetDestinationWorld required on both entrances and exits?  Is SetReceivedPortal required on both entrances and exits?  Any ideas what's going on/what I'm doing wrong?

 

Share this post


Link to post
Share on other sites

Let me see if I can shed some light here:

 

Are IDs unique to a specific shard, or unique across all shards?

 

You are right, these are per-shard.

 

If I'm making a world that requires a while to set up and configure is there a way to start a new game in that properly set-up world?  Alternatively, is there a way to pause time in a server while still being able to move around?

 

Your best bet here would be to make a mod that finds the portals and does the configuration on startup, rather than manually warping around and doing it.

 

Also, c_migrateto(shard_id, 1) sometimes locates me at a mined surface cave entrance, even if it wasn't mined before (not sure if it's creating it or not).

 

If the hole you are "coming out of" was plugged, travelling to it automatically unplugs it.

 

So I figured out the way IDs work even though I don't have a good way of identifying the ID of a particular cave entrance or exit (IDs are unique to a shard, but not across all shards, they start from 1 and increment by 1). 
 

 

If instead of using c_find, you use c_selectnext, it stores the result of the search in c_sel(). You can do things then like c_goto(c_sel()), or print(c_sel().components.worldmigrator.id) to get the portal id.

 

I also don't know if it's intended behavior but c_gonext("cave_entrance") only loops through unmined cave entrances.

 

 

Once it's mined, it turns into cave_entrance_open, so you can find/goto/etc. 

 

6. c_find("cave_exit").components.worldmigrator:SetReceivedPortal(1,11)

 

It looks like you have the parameters flipped, the first one is the "from world", the second one is the "from portal".

 

here's a crappy graph of what I'm talking about:
 

 

This is super cool! However as I'm sure you notice, the "auto linking" isn't doing you any favours.

 

I'm thinking that for the "basic setup" you'll want a mod that does this, to manage the "pairs" of overworlds and caves:

  • Has a config for what world this world's portals should link to. If more than one destination world, it splits the portals among them.
  • On startup, it forces all the portals to point to the correct world, and turns off the auto-linking (described in the top post).

Then a second mod similar to the first with lower priority, just for your scenario, to "cross link" the world pairs:

  • On startup, looks for a portal with a specific ID, and links it to another specific world.

Each of your servers would have a slightly different modoverrides.lua, causing them to link in their own ways.

 

Unfortunately managing the "received" portals isn't simple and we haven't put in a good way of doing this yet. Right now it just assumes that the IDs match on both sides, which is pretty gross. This is enforced in OnLoad, so if you want to overwrite it you have to do it (again, ideally in a mod) one frame after the game loads (e.g. using inst:DoTaskInTime(0, ...)).

 

Hope that answers some of your questions and gets you in the right direction... Just ask if you need more specific advice!

Share this post


Link to post
Share on other sites

Hi,

 

I am trying to setup a Overworld with 2 caves but when the 2rd slave joins either the master gets killed or the 1st slave gets killed, like this:

About to start a shard with these settings:  ShardName: CavesA  ShardID: 1000  ShardRole: SLAVE  MasterHost: 127.0.0.1  MasterBind: (null)  MasterPort: 11001[SHARD] Connecting to master...Telling Client our new session identifier: 78D370CB17D4D87DSim paused[Steam] SteamGameServer_Init (11015, 11010, 11016)[S_API FAIL] SteamAPI_Init() failed; SteamAPI_IsSteamRunning() failed.Setting breakpad minidump AppID = 322330[Steam] SteamGameServer_Init Success[SHARD] Sending slave information to master...[SHARD] Registering in lobby as a slave server...[SHARD] Slave is now ready![SHARD] Registering in lobby as a slave server...World '1' is now connected	Killed

Any ideas of what might be happening?

 

Thanks for your help.

 

ConfigurationsAndLogs.zip

Share this post


Link to post
Share on other sites

Klei somewhere stated that they've integrated automatic sinkhole/portal linking with the tree-world and the cave-world - maybe they don't support linking to two cave-worlds. Did you try running only one cave-world? Your settings.ini files look good, I believe.

Share this post


Link to post
Share on other sites

Klei somewhere stated that they've integrated automatic sinkhole/portal linking with the tree-world and the cave-world - maybe they don't support linking to two cave-worlds. Did you try running only one cave-world? Your settings.ini files look good, I believe.

If I don't start the 2nd caves server it works great.

 

What is the command to disable the automatic sinkhole/portal linking?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now