"Failed to save file" Error causing server fail

[Hedwig 0]

Recently I've been dealing with "Failed to save file" error on my servers. This happens frequently since April 2022, with error logs like this:

————————————————————————————————

[03:26:00]: Available disk space for save files: 45920 MB
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7H31E0AB76Q/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7H31E0AB76Q/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeicfjlhlijfirjpfvnmndghgflgkoqcgginknnmmeijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7H31E0AB76Q/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeicfjlhlijfirjpfvnmndghgflgkoqcgginknnmmeijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7MGS7K25K6H/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7MGS7K25K6H/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeihfjllkijhcrjnmvnxmdgihflnloqodginknnvleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7MGS7K25K6H/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeihfjllkijhcrjnmvnxmdgihflnloqodginknnvleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7IEMRM7FCSL/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7IEMRM7FCSL/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeidfjlnkijjbrjqgvnrmdgfhflkkoqgdgiqinnrleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7IEMRM7FCSL/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeidfjlnkijjbrjqgvnrmdgfhflkkoqgdgiqinnrleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7I4F075E1SF/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7I4F075E1SF/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeidfjlmlijcbrjsmvnlndgdhflnkoqeggiqinnlleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7I4F075E1SF/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeidfjlmlijcbrjsmvnlndgdhflnkoqeggiqinnlleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7G3IPRNTENN/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7G3IPRNTENN/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeirfjlhlijnbrjsgvnoldgmgflmjoqadgifhnntleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7G3IPRNTENN/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeirfjlhlijnbrjsgvnoldgmgflmjoqadgifhnntleijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Serializing user: session/636276601E1A1D5C/A7JJE863FU7Q/0000000034
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7JJE863FU7Q/0000000034.meta
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeiefjlokijbbrjkmvnkndgjhflkkoqaegimknnmmeijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijlsmijjdrjphvnijdghefl.
[03:26:00]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/A7JJE863FU7Q/0000000034
[03:26:00]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgishnnkoeiefjlokijbbrjkmvnkndgjhflkkoqaegimknnmmeijhjlilijeirjsmvnmndgghfliloqfggitknnooeioijl.
[03:26:00]: Missing reference:    114107 - beefaloherd    ->    102530    102530 - beefalo    
[03:26:00]: Missing reference:    111385 - terrarium    ->    176851    nil    
[03:26:01]: Serializing world: session/636276601E1A1D5C/0000000034
[03:26:01]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/0000000034.meta
[03:26:01]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgitknnnoeikijlilijeirjsmvnmndgghflhloqbggifjnnqjeipdjlmhijfdrj.
[03:26:01]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/shardindex_time
[03:26:01]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnukdgheflghoqbbgikfnntjeiodjlniijmerjfgvnijdgpeflvioqabgi.
[03:26:01]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/shardindex
[03:26:01]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnukdgheflghoqbbgikfnntjeiodjlniijmerj.
[03:26:01]: [CRITICAL] Failed to save file: APP:Klei//DoNotStarveTogether/Cluster_9/Master/save/session/636276601E1A1D5C/0000000034
[03:26:01]: [CRITICAL] Fingerprint:hbrjimvnqldgdfflhhoqabgirgnnokeiggjljkijadrjhhvnvkdgmeflrioqgcgipgnnpkeildjlmhijldrjqivnqldgcgfltioqgbgiognnqjeipdjlsiijaerjrivntodgrgfluioqabgikfnnsneijhjlmkijldrjefvntkdgcdflhjoqbcgisfnnpkeiqejlniijacrjfhvnlkdgdeflmhoqnbgiofnnpkeijhjlhkijidrjpivnpjdgcdflnioqhcgieinnuoeijhjlvkijfdrjrivnijdgdeflghoqkfgiqgnnmjeiqejlniijlhrjrivnjkdgjdflhhoqmbgiefnntjeijhjlklijhirjmmvnondgfhflkloqdggitknnmoeipfjljlijfbrjtmvnimdgdhflhkoqkfgitknnnoeikijlilijeirjsmvnmndgghflhloqbggi.

————————————————————————————————

When this happens, the server could not save the save files everyday morning, and when a rollback() occurs, the server will automatically regenerate and crash during the regeneration.

The servers have been running for more than a year, and this error started to occur this month without any sign. There's plenty of room in the diskspace, and the error has occured in three different cloud servers so it couldn't be the problem of the system. This indeed bothers me and it would be appreciated if anyone could look into and tell how to solve such problems.

(System: Windows Server 2019)

server_log_2022-04-06-20-09-17.txt

[JJJoker 0]

There may be a problem with the disk. Try rebuilding the cloud disk or replace with a new cloud disk. If it is a physical disk, try to check the disk for bad sectors, or replace the disk with a new one.

[Hedwig 0]

30 minutes ago, JJJoker said:

There may be a problem with the disk. Try rebuilding the cloud disk or replace with a new cloud disk.

According to the service manual, the cloud servers use distributed storage systems so that there shouldn't be disk failures, but thanks^-^ I'll check for that.

[nome 6910]

Is this on Linux? An strace might be informative as well.

[Hedwig 0]

14 hours ago, nome said:

Is this on Linux? An strace might be informative as well.

It's on Windows Server

Maybe I should try similar tools such as process monitor?

[gznc 24]

I hosted many worlds in a machine but master shard always meets this error.

I meet this since April as you.

[Hedwig 0]

On 5/8/2022 at 3:49 AM, gznc said:

I meet this since April as you.

你有头绪么，我现在完全不知道什么问题

[Axsdenied 0]

Cloud servers using distributed storage are not immune to data problems.  Assuming permissions have not changed (i.e. the server process still has access to that file/dir) and no other process is locking that file/directory for writing, I would very much suspect the storage it's associated with.

Are you able to move that server save to a different disk just for testing?

[Hedwig 0]

2 hours ago, Axsdenied said:

Cloud servers using distributed storage are not immune to data problems.  Assuming permissions have not changed (i.e. the server process still has access to that file/dir) and no other process is locking that file/directory for writing, I would very much suspect the storage it's associated with.

Are you able to move that server save to a different disk just for testing?

This problem has been happening in at least 3 different cloud platforms (from 2 seperate cloud services) since the same time at early April, which make me hardly consider it a storage system problem. However, servers running on local machines haven't experience this yet.

I'm now running a process monitor on one of the cloud platforms and hope it could find something.

Edited May 11 by Hedwig

[gznc 24]

已经基本查明了，句柄泄露

[Hedwig 0]

12 hours ago, gznc said:

已经基本查明了，句柄泄露