Johnny Rollback To March

[Krydos]

It looks like during the raid rebuild the Johnny server data rolled back to March 8th, so we'll only be able to provide backups from data prior to that date. You'll need to rely on your own personal backups that you made over the last few months for any changes since that date. After we're done backing up the data that is available we'll be wiping the server, and rebuilding Johnny.

[wygnany852225233]

But what about accounts created after March 8th? Mine was created at May 20th and now i see only "You have entered incorrect username or password." and no email after using "Forgot your password?" option.

[wolstech]

Accounts created after March 8 were lost in their entirety. All other accounts (created before that date) have rolled back to the condition they were in on March 8. Files and databases will have reverted, domains that were added have been removed, etc.

That said, there are a few exceptions to the March 8th date. If your account appears in the below list, a backup of the account as of the backup_date should be available via https://heliohost.org/backup/ 

+------------------+---------------------+---------------------+
| username         | backup_date         | signed_up           |
+------------------+---------------------+---------------------+
| deknoinarak      | 2024-05-11 18:37:35 | 2022-12-27 00:01:08 |
| howl             | 2024-05-05 17:50:16 | 2024-05-05 18:07:26 |
| iadjali46        | 2024-05-09 17:49:24 | 2024-05-10 22:10:05 |
| tymek805         | 2024-05-14 13:07:16 | 2024-05-14 13:45:59 |
| beoutq586        | 2024-06-12 16:51:18 | 2024-06-13 10:38:22 |
| kokofixcomputers | 2024-05-15 19:50:46 | 2024-05-15 19:55:57 |
| szhiro           | 2024-05-24 19:00:25 | 2024-05-24 20:18:21 |
| dlane            | 2024-07-14 20:02:56 | 2024-04-19 00:03:51 |
| websys           | 2024-05-23 13:49:34 | 2024-05-02 12:02:33 |
| forgeekdoing     | 2024-05-14 15:08:58 | 2024-05-08 00:09:49 |
| ewanireone       | 2024-06-12 15:04:52 | 2024-06-12 15:10:20 |
| fastupload       | 2024-06-24 13:26:48 | 2024-06-24 00:15:35 |
| abdualghurarh    | 2024-07-02 21:39:40 | 2024-06-25 00:06:32 |
| vectr            | 2024-06-30 14:33:38 | 2024-06-27 12:07:54 |
| olenamail76      | 2024-07-06 15:33:51 | 2024-07-03 00:02:28 |
| whitedevil       | 2024-07-06 18:23:12 | 2024-07-06 18:27:06 |
+------------------+---------------------+---------------------+
16 rows in set (0.02 sec)

 

[nmbayu]

After backup what I'm going to do next? 

[miniurl]

After rebuild of Jhonny, all the accounts will be pre created or we will get invite to create account on Jhonny, as happened previously?

[Krydos]

On 7/27/2024 at 12:12 AM, nmbayu said:

After backup what I'm going to do next? 

We will be rebuilding Johnny, and then you can restore your backup on to the new Johnny. If you don't want to wait for that to happen you can donate $1 to immediately get an account on Tommy, or you can get a VPS immediately for $4 per month. We will also be releasing the Morty server soon as well.

8 hours ago, miniurl said:

After rebuild of Jhonny, all the accounts will be pre created or we will get invite to create account on Jhonny, as happened previously?

After Johnny is rebuilt, if you try to login or reset the password of an account that was on Johnny, but was deleted due to the rollback or the rebuild, you will be sent an invite to recreate your account without needing to contact an admin, or wait until the free signup period at midnight or noon UTC.

[azubriy]

Are the databases of Johnny-accounts also corrupted?

[Krydos]

Since you created your account after March 8th there is no data for you on the server. You'll have to rely on the backups that you made yourself.

[Ipsum]

16 hours ago, Krydos said:

Since you created your account after March 8th there is no data for you on the server. You'll have to rely on the backups that you made yourself.

One problem I notice is that something is wrong with the proxy on the Apache server and a 502 error code is returned when trying to access the site

[Krydos]

3 minutes ago, Ipsum said:

One problem I notice is that something is wrong with the proxy on the Apache server and a 502 error code is returned when trying to access the site

Yep literally everyone else has noticed it too. That's why we mentioned it in a news post.  

On 7/25/2024 at 12:28 AM, Krydos said:

Apache won't boot up do to thousands of errors in the configuration.

 

[miniurl]

Thanks Krydos and team. Hope everything goes well.

[rvb]

Can I ask how often is this happening? I mean, rebuilding entire Johny server with wiping out new accounts? 

On 7/28/2024 at 9:12 AM, miniurl said:

, as happened previously?

Maybe I should better go for Tommy?

[wolstech]

We have some form of either scheduled rebuild or a disaster requiring a rebuild every 2-4 years it seems.

This is I believe the third or fourth time we've had a disk failure in my 11+ years, and the first where we've just flat out lost a bunch of accounts, though in past failures the accounts were often recovered in pieces that didn't necessarily restore cleanly (e.g. plain zip archives instead of a self-restoring archive, and I think we lost some databases one time due to InnoDB crashing). This is also the first disk failure for a Plesk based server.

There were some lessons learned here, because this failure occurred on a server with disks that actually had redundancy. In all prior incidents there was no redundant storage. This time around we had a mirror volume, but also a policy of disconnecting the idrac management interfaces on our servers for security reasons (that decision was a relic of the days when we didn't have redundancy and as such had no need for them). As such we had no way of knowing one of the two disks in the mirror had failed, so it was never replaced before the second one also failed. That policy has since been changed as a result of this incident. 

 

It's worth noting that we rebuild servers every 3-4 years anyway for preventative maintenance. Even with no failure, users would likely see their account either moved to another server, or archived while the server is rebuilt, then unarchived again upon completion. The servers tend to develop strange issues just due to clutter from account churn over time even if they don't outright fail, and software needs updating too (the OS especially, and they don't always update gracefully) so its easier to rebuild them to ensure they're clean and current.

Once Johnny is fixed and Morty is out the door, Tommy will likely be up for a rebuild to update it since its OS will be obsolete soon, and it has a lot of weird quirks that developed as a result of recovering it from ransomware incident in early 2023...