Jump to content

Is login server down now?


Recommended Posts

  • Developer
On 10/01/2019 at 12:48 AM, iBrocca said:

Just wondering what caused the problem?

We had a process that was upgrading older accounts to a new format in the background. It was set to a very low priority and @Diabu watched it carefully for the first few days to make sure it wasn't impacting users. Sadly it appears that it was running just an infinitesimal smidgen over the capacity we had to spare, causing secondary indexes to fall behind just a little each day. Finally that exceeded a limit and requests to those indexes started to fail. To make it worse, since it happened in the middle of the night and nobody had done anything dangerous lately nobody was actively watching it, and our automated systems did not catch it because sadness. :(

Luckily  @YormanA happened to notice it and brought it to my attention, I disabled the background process and we then waited for the secondary indexes to catch up. Big thanks to @Zillvr who pushed the emergency-wake-up-nome button! Sadly it was a few minutes after I saw Yorman's message but had that not been the case he'd have been the guy that saved us all from a whole night of DST downtime!

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

Please be aware that the content of this thread may be outdated and no longer applicable.

×
  • Create New...