1/8/19 - Site Outage

[Dragoneer]

We're currently investigating an issue which is causing unexpected downtime (502 errors).

 

[Dragoneer]

We believe we've identified the source which caused the unexpected outage, and had to reboot one of our servers. The server is currently going through routine checks. We'll update you once we know more.

 

[yak]

Thumbnail server has experienced filesystem corruption and is currently running foreground filesystem checks.
Even if some of the files end up missing because of this it's not going to be an issue due to missing thumbnails being automatically regenerated by the system.
Unfortunately a filesystem check of 5TB of data on a 14TB partition is going to take a while to complete even on server grade hardware and as such we currently do not have estimates for when the site will be back online.

 

[yak]

First run of filesystem check completed. Three files and one directory have been corrupted which like I said earlier, isn't a big deal.
Additional run is required and is now in progress.

 

[yak]

Just an update, the filesystem checks are still running.

 

[Dragoneer]

We're taking time to do some necessary server updates, then we'll be bringing up the site. We'll post once we're back up and running.

 

[yak]

We are finishing up with the operating system updates.