Hedwig

"Failed to save file" Error causing server fail

Recommended Posts

Hedwig    0

Recently I've been dealing with "Failed to save file" error on my servers. This happens frequently since April 2022, with error logs like this:

————————————————————————————————

[03:26:00]: Available disk space for save files: 45920 MB
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7H31E0AB76Q/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7H31E0AB76Q/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeicfjlhlijfirjpfvnmndghgflgkoqcgginknnmmeijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7H31E0AB76Q/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeicfjlhlijfirjpfvnmndghgflgkoqcgginknnmmeijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7MGS7K25K6H/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7MGS7K25K6H/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeihfjllkijhcrjnmvnxmdgihflnloqodginknnvleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7MGS7K25K6H/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeihfjllkijhcrjnmvnxmdgihflnloqodginknnvleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7IEMRM7FCSL/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7IEMRM7FCSL/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeidfjlnkijjbrjqgvnrmdgfhflkkoqgdgiqinnrleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7IEMRM7FCSL/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeidfjlnkijjbrjqgvnrmdgfhflkkoqgdgiqinnrleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7I4F075E1SF/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7I4F075E1SF/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeidfjlmlijcbrjsmvnlndgdhflnkoqeggiqinnlleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7I4F075E1SF/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeidfjlmlijcbrjsmvnlndgdhflnkoqeggiqinnlleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7G3IPRNTENN/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7G3IPRNTENN/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeirfjlhlijnbrjsgvnoldgmgflmjoqadgifhnntleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7G3IPRNTENN/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeirfjlhlijnbrjsgvnoldgmgflmjoqadgifhnntleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7JJE863FU7Q/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7JJE863FU7Q/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeiefjlokijbbrjkmvnkndgjhflkkoqaegimknnmmeijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7JJE863FU7Q/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeiefjlokijbbrjkmvnkndgjhflkkoqaegimknnmmeijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Missing reference:    114107 - beefaloherd    ->    102530    102530 - beefalo    
[03:26:00]: Missing reference:    111385 - terrarium    ->    176851    nil    
[03:26:01]: Serializing world: session/636276601E1A1D5C/0000000034
[03:26:01]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/0000000034.meta
[03:26:01]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgitknnnoeikijlilijeirjsmvnmndgghflhloqbggifjnnqjeipdjlmhijfdrj.
[03:26:01]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/shardindex_time
[03:26:01]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnukdgheflghoqbbgikfnntjeiodjlniijmerjfgvnijdgpeflvioqabgi.
[03:26:01]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/shardindex
[03:26:01]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnukdgheflghoqbbgikfnntjeiodjlniijmerj.
[03:26:01]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/0000000034
[03:26:01]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgitknnnoeikijlilijeirjsmvnmndgghflhloqbggi.

————————————————————————————————

When this happens, the server could not save the save files everyday morning, and when a rollback() occurs, the server will automatically regenerate and crash during the regeneration.

The servers have been running for more than a year, and this error started to occur this month without any sign. There's plenty of room in the diskspace, and the error has occured in three different cloud servers so it couldn't be the problem of the system. This indeed bothers me and it would be appreciated if anyone could look into and tell how to solve such problems.

(System: Windows Server 2019)

server_log_2022-04-06-20-09-17.txt

Share this post


Link to post
Share on other sites
JJJoker    0

There may be a problem with the disk. Try rebuilding the cloud disk or replace with a new cloud disk. If it is a physical disk, try to check the disk for bad sectors, or replace the disk with a new one.

Share this post


Link to post
Share on other sites
Hedwig    0
30 minutes ago, JJJoker said:

There may be a problem with the disk. Try rebuilding the cloud disk or replace with a new cloud disk.

According to the service manual, the cloud servers use distributed storage systems so that there shouldn't be disk failures, but thanks^-^ I'll check for that.

Share this post


Link to post
Share on other sites
Hedwig    0
14 hours ago, nome said:

Is this on Linux? An strace might be informative as well.

It's on Windows Server :(

Maybe I should try similar tools such as process monitor?

Share this post


Link to post
Share on other sites
gznc    24

I hosted many worlds in a machine but master shard always meets this error.

I meet this since April as you.

Share this post


Link to post
Share on other sites
Hedwig    0
On 5/8/2022 at 3:49 AM, gznc said:

I meet this since April as you.

你有头绪么,我现在完全不知道什么问题

Share this post


Link to post
Share on other sites
Axsdenied    0

Cloud servers using distributed storage are not immune to data problems.  Assuming permissions have not changed (i.e. the server process still has access to that file/dir) and no other process is locking that file/directory for writing, I would very much suspect the storage it's associated with.

Are you able to move that server save to a different disk just for testing?

Share this post


Link to post
Share on other sites
Hedwig    0
2 hours ago, Axsdenied said:

Cloud servers using distributed storage are not immune to data problems.  Assuming permissions have not changed (i.e. the server process still has access to that file/dir) and no other process is locking that file/directory for writing, I would very much suspect the storage it's associated with.

Are you able to move that server save to a different disk just for testing?

This problem has been happening in at least 3 different cloud platforms (from 2 seperate cloud services) since the same time at early April, which make me hardly consider it a storage system problem. However, servers running on local machines haven't experience this yet.

I'm now running a process monitor on one of the cloud platforms and hope it could find something.

Edited by Hedwig

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now