wolstech

Krydos can you take a look at this? /home1 seems to have unmounted due to the attack (or more likely the mitigation of the attack) yesterday. All users with a /home1 folder are seeing "an error occurred processing this directive" in cPanel, and a 403 error on their website...

I figured out the 403 errors, now we need Krydos to fix them. The issue is related to /home1 being unavailable on Tommy. For the unfamiliar, /home1 is an additional home partition that's stored on our NAS, it was added to increase disk capacity a while back. I picked several additional sites beyond the 10 I tried earlier...all users with a /home1/<username> home folder are affected.

The /home1 share that points to our NAS is offline. Escalating to Krydos.

Your logins are definitely not counting for whatever reason. I moved your last login date ahead one month, so your last login date is now in the future (8/11). Use https://tommy.heliohost.org:2083/ (with nothing after the slash) to log in going forward and we'll see if it makes a difference. I've seen a few users with this issue now, and they always were having the problem due to using our website to log in. Using cPanel directly to log in fixed it for them.

I think this is to do with the IP changes Krydos made...lets see what he says. Looking into this further shows that is issue is related to /home1 being unavailable. /home1 is the home folders stored on our NAS. I picked several sites, and all of them kept on /home1 are affected.

@alein: I think the IP changes last night broke something. Your domain isn't even resolving, let alone working. I made you a topic here: https://www.helionet.org/index/topic/33852-cpanel-not-working-domain-not-resolving/

Can you create a separate topic for the forbidden error? You two the only ones I can find experiencing it. I picked 10 random websites on Tommy and they all loaded properly. The server does have a few configuration issues though, for example the main tommy.heliohost.org is still pointed to my dedicated IP right now...(this was how we kept cPanel working during the outage).

It's a botnet (or someone else with a large and random IP pool), not a single source. If we block one IP, the attack just comes from elsewhere...

Rax software / rax2 is my account. Not harmful, it just looks strange with my IP hosting the main server domain. The forbidden errors are related too (accounts that were pointed to my ip for offloading may show this since they're not allowed to be hosted there.) Krydos moved a bunch of things to my dedicated ip to mitigate the ddos and keep services like cpanel available during the attack. The side effect of that is that certain pages now have no idea what to show and are showing one of my websites instead because it's supposed to be a dedicated ip. That should go away when Krydos finishes whatever he's doing.

We're aware. The attack is still ongoing. The old shared IP is intentionally down while we work to mitigate it. See https://www.helionet.org/index/topic/33842-2018-08-07/ for more information...

Please see https://www.helionet.org/index/topic/33812-johnny-maintenance/

This account has a compromised CMS installation that has been affected by the recent AnonymousFox hack and cannot be unsuspended. An invitation will be sent to you shortly so you can create a new account. Please restore your data using a backup. The data from your old account cannot be recovered or returned to you due to the possibility of the account having been used for Phishing. As a reminder, when selecting a CMS, we highly recommend that users not use WordPress. WP and it's extensions are notorious for having security issues such as the one you (and everyone else on Tommy) experienced, and it has the worst security track record of any CMS out there. Using a different program will help prevent this from happening again.

You already have an account called mrshm25 on Ricky that needs to be deleted before you can sign up again. You're only allowed one account and creating a second one violates our Terms of Service. In addition to that, the creation failed because of: The domain mrshah.heliohost.org has been cleaned up. ...and finally, on top of all that, the server is down anyway pending a migration to a new IP to mitigate a DDoS attack.

It's null routed right now. Also, if you see the block on the monitor where Ricky was being slow, that was due to network congestion on our switch (Ricky wasn't targeted, Tommy was).

Ziad's got it, but to clarify the reason above a bit more. The Tommy attack is actually a different type of DDoS (wasting traffic vs. just leaving connections open all day on Johnny). It may or may not be the same attacker. It could be retaliation for cleaning up AnonymousFox too. The people who have the resources to launch these attacks are almost certainly involved in other cybercrime too. You don't generally just keep a botnet around (or hire one and keep it on retainer) for no reason. Spam and phishing have the one of the best effort-to-profit ratios out there. Both are relatively easy to execute and very profitable when they pay off (send 1000 phishing emails, if just one user falls for it they can potentially get a few grand from a paypal account...). Sometimes other abuse comes along, usually with phishing or spam being the end goal. For example look at AnonymousFox, who used a 0-day WP exploit to take over more than 100 websites, saving him the hassle of getting domains and accounts and giving him the advantage of the site owner's reputation. Many of those then had a phishing site and/or spambot uploaded. Also, most hosting companies are reactive. Phisher goes phishing, email and security companies squawk, and eventually the host gets abuse reports and bans them. We don't wait for that report and instead proactively monitor for abuse. To cyber-criminals, that means "We make easy money difficult." Their response to us wasting their time is a botnet to the face.

Tommy has returned to normal. The complete outage of Tommy's public-facing services was actually caused by a (much more drastic) mitigation used on Tommy, not the attack itself. Unlike Johnny, where we just allow the attack to subside (well...hope it subsides), Tommy shares the Eddie hardware with Cody, which is responsible for our website and forums, admin tools, and also provides a name server. To avoid the load from the attack bogging Cody down, we had our provider intentionally null route Tommy so the attack couldn't reach the server. The downside is all legitimate traffic also goes off into the void when this is done.

Tommy has returned to normal.

I got a response from Krydos on this. Turns out his botnet is big enough, we just blocked the shared IP to keep it from being effective. Bad news is that means all the websites are down. The good news is all the other stuff is on another IP, which is why cPanel and everything else is accessible.

The mail server itself seems to be working fine (I can telnet to it inside SSH and get the expected responses, webmail also works). Apache is actually up as well if I telnet to it from localhost and request a document. The issue is that the traffic can't get in or out right now. EDIT: Just heard for Krydos...it's blocked intentionally to mitigate the attack.

I think there's some stuff Krydos can try, but we're gonna see if it subsides first most likely. We're not even sure it's the same attacker, but it's reasonable to believe so considering the attack started within 24 hours of Johnny being pulled for maintenance. Good news is that Tommy is beefy enough that he doesn't just collapse from the load caused by Apache and the firewall trying to block it. Apache is overwhelmed by the botnet, but everything else on him should be working just fine. cPanel, FTP, and email are up. Just the actual web server that isn't. Johnny on the other hand couldn't handle the load and basically folded under pressure.

Your account was suspended for causing high server load. I have unsuspended your account, but please try to limit the load you put on our servers as it slows down not only your site, but the sites of all other HelioHost users sharing your server. If you still see the suspended page, please clear your cache. If you need help figuring out why your site is causing such high load let us know and we can try to help. If the high load is simply because your site is getting a lot of traffic you might consider trying paid hosting from our partner starting at only a cent for the first month. https://www.heliohost.org/partners/hostgator

This account has a compromised CMS installation that has been affected by the recent AnonymousFox hack and cannot be unsuspended. An invitation will be sent to you shortly so you can create a new account. Please restore your data using a backup. The data from your old account cannot be recovered or returned to you due to the possibility of the account having been used for Phishing. As a reminder, when selecting a CMS, we highly recommend that users not use WordPress. WP and it's extensions are notorious for having security issues such as the one you (and everyone else on Tommy) experienced, and it has the worst security track record of any CMS out there. Using a different program will help prevent this from happening again.

This account has a compromised CMS installation that has been affected by the recent AnonymousFox hack and cannot be unsuspended. An invitation will be sent to you shortly so you can create a new account. Please restore your data using a backup. The data from your old account cannot be recovered or returned to you due to the possibility of the account having been used for Phishing. As a reminder, when selecting a CMS, we highly recommend that users not use WordPress. WP and it's extensions are notorious for having security issues such as the one you (and everyone else on Tommy) experienced, and it has the worst security track record of any CMS out there. Using a different program will help prevent this from happening again.

Considering we can't fix the ddos on Johnny, we have to just wait it out, which considering this botnet's persistence, it could be months. Maybe it will move back to Johnny when that maintenance is completed, but who knows. More than likely the goal of the attacker is to crash all of our servers and run us out of business. The attack started shortly after we banned ~150 phishing sites that signed up over the course of a week (mostly paypal phishing, and significantly more than normal), so my theory is that its retaliation for us banning some clown who wanted to phish paypal accounts. That's because the attack just overloads Apache, effectively keeping it from doing anything. The cPanel stuff runs under a dedicated web server application known as cpsrvd that is unaffected. Email and the like should still work too. Unlike Johnny, which buckled from load, Tommy is much beefier, so between him simply having a lot more capability and his firewall not hogging the CPU, he handles being attacked much better. TL;DR: Some idiot phisher is attacking us and his botnet ain't big enough

Yes it is. Supposedly the ddos from Johnny moved there because we took Johnny down for maintenance...