[Solved] Website is frequently down

[xcrow]

I'm on Tommy plan, and I've noticed that the website is frequently down more than it should. At first I brushed it off since the plan does include some down time
but this is definitely a lot it happens every few minutes and it's making me start to wonder whether there's something wrong with my account specifically 

[wolstech]

What errors are you seeing?

Error 503 was a known problem on this server recently (should be resolved now), and 502 for a few minutes every 2 hours is normal (Apache restart).

[rexkh]

It happens everyday once if not a few. I'm curious why Apache always give 5xx error

[wolstech]

Please see https://wiki.helionet.org/502_Bad_Gateway for information on this error.

[rexkh]

I've played around with "systemctl restart apache2". For my experience, it shouldn't take more than a few seconds. It really take a few minutes to complete in Johnny and Tommy? I'm just curious.

[wolstech]

Yes, it does take that long in our case, because there's several thousand domains hosted on one server, which means a ton of Apache vhosts.

Johnny and Tommy each have about 5500 vhosts (a typical domain uses 2 vhosts, one for plain HTTP and one for HTTPS). Your typical VPS is probably going to have like 10-20 vhosts. An instance of apache on your home computer for localhost development likely has 1 or 2.

Then there's the fact that the servers are also quite busy, so load is not always the lowest...

[rexkh]

That's a lot I guess. I think 5xx error cause by  apache2 crash. What make it crash in the first place? Out of Memory issue?

[wolstech]

It's not crashing. We're forcing it to restart intentionally. The errors occur because nginx is used in front of Apache as a proxy, and it spits out that error when it can't reach apache.

[rexkh]

I'm aware that Johnny restart apache2 every 2 hours to load new vhost so 24h it should do it 12 times but in incident report, down time is only around 5-6 times a day. It doesn't make sense to me. Do I miss some parts of the story?

[wolstech]

Just the monitor not catching it. The monitor checks on an interval, and if happens to occur between checks, it simply never shows on the monitor. Server load also affects the speed at which it restarts, so during times of low load, the restart completes faster, resulting in shorter error duration and an increased likelihood of it being missed by the monitor.

And finally, the 2 hours thing isn't a hard 2 hours. It's at least 2 hours between restarts. There's times where that restart is closer to 3 hours because the server delays maintenance tasks if it's experiencing high load. The background process we developed to handle routine server maintenance (including apache restarts) is designed to only do its work when server load is below a set threshold.