Leaderboard

We have some form of either scheduled rebuild or a disaster requiring a rebuild every 2-4 years it seems. This is I believe the third or fourth time we've had a disk failure in my 11+ years, and the first where we've just flat out lost a bunch of accounts, though in past failures the accounts were often recovered in pieces that didn't necessarily restore cleanly (e.g. plain zip archives instead of a self-restoring archive, and I think we lost some databases one time due to InnoDB crashing). This is also the first disk failure for a Plesk based server. There were some lessons learned here, because this failure occurred on a server with disks that actually had redundancy. In all prior incidents there was no redundant storage. This time around we had a mirror volume, but also a policy of disconnecting the idrac management interfaces on our servers for security reasons (that decision was a relic of the days when we didn't have redundancy and as such had no need for them). As such we had no way of knowing one of the two disks in the mirror had failed, so it was never replaced before the second one also failed. That policy has since been changed as a result of this incident. It's worth noting that we rebuild servers every 3-4 years anyway for preventative maintenance. Even with no failure, users would likely see their account either moved to another server, or archived while the server is rebuilt, then unarchived again upon completion. The servers tend to develop strange issues just due to clutter from account churn over time even if they don't outright fail, and software needs updating too (the OS especially, and they don't always update gracefully) so its easier to rebuild them to ensure they're clean and current. Once Johnny is fixed and Morty is out the door, Tommy will likely be up for a rebuild to update it since its OS will be obsolete soon, and it has a lot of weird quirks that developed as a result of recovering it from ransomware incident in early 2023...