Severe Load And Mysql Time Out Occurring On Stevie

[n30n]

Hello,

 

Whats with the severe load on Stevie since 8 AM UTC? Also seeing this error

 

"Database Error: Unable to connect to the database:Could not connect to MySQL"

 

much more frequently now. I am not the mod so I don't know the reality behind it (might be MYSQL server is getting abused) although this is the similar kind of stuff that happened before the Stevie went down for 4,5 days last time. Is anybody keeping an eye on it right now?

 

Regards,

 

Navaid Farooqui

[maths222]

I don't know why there is so much load, but there is a lot more disk swap memory being used than there was yesterday. (75% vs. 1%)

[Krydos]

I suspended a couple accounts that were causing a lot of load on mysql and the server monitor indicates that everything is returning to normal now. The swap usage does seem rather high, but maybe that will drop with time too.

[Guest xaav]

It appears apache is eating up a huge chunk of memory, causing the slowness.

[n30n]

what version of apache are we using on Stevie???

[Krydos]

Apache version 2.2.16

[n30n]

Ok.... so you said above that some of the accounts were causing load on MYSQL. Swap file is large too and according to XAAV apache is eating up a huge chunk of memory causing the slowness....

 

so just curious

 

http://httpd.apache.org/security/vulnerabilities_22.html

 

especially quoting

A flaw was found in the way the Apache HTTP Server handled Range HTTP headers. A remote attacker could use this flaw to cause httpd to use an excessive amount of memory and CPU time via HTTP requests with a specially-crafted Range header. This could be used in a denial of service attack.

Advisory: CVE-2011-3192.txt Issue public: 20th August 2011

Update released: 30th August 2011 Affected: 2.2.19, 2.2.18, 2.2.17, 2.2.16, 2.2.15, 2.2.14, 2.2.13, 2.2.12, 2.2.11, 2.2.10, 2.2.9, 2.2.8, 2.2.6, 2.2.5, 2.2.4, 2.2.3, 2.2.2, 2.2.0

 

Can this be the cause? or is it just casual MYSQL load increase?

[Krydos]

It could be. The symptoms certainly seem to fit.

 

However, I suspended two accounts today that were each running really convoluted mysql queries that took ridiculous amounts of time to process repeatedly, and within minutes of those accounts being suspended the server load dropped to a reasonable level and everything started working again, so I'm inclined to believe that it wasn't just a coincidence.

 

We do know that apache needs to be upgraded, but as you can probably imagine, it's a rather complicated procedure and stevie already has several more critical issues that should probably be fixed first.

[Guest xaav]

I lowered the MaxRequestsPerChild directive to hopefully eliminate the memory leak.

[n30n]

hmm... All lime all stable now :-) ...

[Ice IT Support]

I am still receiving the "Too many connections" error. I refreshed cache, etc. But at the time of posting, the server monitor stated some recent (short) downtime.

[Guest xaav]

MySQL is being upgraded due to some recent complaints of corruption.

[n30n]

That's odd. I am receiving 500 Internal Server Error. Though I can see my database (Thank God ) . Also there is NO INFO for server load of Stevie in Server Monitor. Is it due to MySQL upgrade. If it is can you tell me whats the approximate time for completion??

[demizide]

That's odd. I am receiving 500 Internal Server Error. Though I can see my database (Thank God ) . Also there is NO INFO for server load of Stevie in Server Monitor. Is it due to MySQL upgrade. If it is can you tell me whats the approximate time for completion??

How long have you seen the 500 page? Because sometimes when I upload a file, I get a 500 when trying to reach it for a couple of minutes, then it works.

 

Edit: the server monitor shouldn't be down because of the upgrade, since it's hosted somewhere else, but it's weird that only the server load on stevie is down, nothing else is..

[n30n]

Actually I am seeing the page right now. Yes it happens occasionally when the load on Stevie is High. I think the only reason why only the server load is down on monitor is that we have the monitors for FTP HTTP and CPANEL. They all should be working fine even if MySQL is down. But there can't be any (or maybe negligible) load on server since most of the sites on Stevie are PHP based.

 

Well, at the heliohost I can see the cross in-front of Stevie. Means its really down. Although I was waiting for a reply from any of the mod about the expected completion time. I don't know whether it takes that long to upgrade MySQL (I think it shouldn't). Maybe somebody just messed it up a bit