[Solved] My VPS is high (on load) again

[Amoss]

So again my VPS gets high loading averages, don't know what is it this time.

I pm Krydos with information and screenshots, please let him know as soon as possible as it happens much too often.

 

Thanks

 

[wolstech]

He'll see this when he next checks the escalated section

[Amoss]

My server is dead for a long time, VPS VPS but it gets spikes all over all the time and nothing can reach it, please someone contact Krydos?

I guess he is not aware but my server is useless, no one can't work on it.

I took a VPS because I was told its uptime is great and nothing will interfere it, until now, it seems the opposite, he is dead often because of high loads.

Edited June 6, 2019 by Amoss

[Krydos]

I replied on the PM. Sorry, you're the only person in the last 10 years that I've tried to provide support through PMs. As long as you don't mind some of your log files and stuff being public it would be easier for me to provide support through regular forum posts. Also you get the benefit of Wolstech or someone else chiming in with possibly useful information.

[Amoss]

The problem is still happening so a new PM was sent with info and screenshots. Only then I see this reply.

If it's ok with you, please read the PM, reply there and we'll continue here the next time.

 

Thanks

Edited June 7, 2019 by Amoss

[Amoss]

After trying to redeploy my war, everything is a mess, nothing works and I'm lost.

I'm just left with the hope that at some point Krydos will see this...

[Krydos]

After trying to redeploy my war, everything is a mess, nothing works and I'm lost.

I'm just left with the hope that at some point Krydos will see this...

It's working for me https://stockdiv.com/Stockdiv/Stockdiv

[Sn1F3rt]

 

After trying to redeploy my war, everything is a mess, nothing works and I'm lost.

I'm just left with the hope that at some point Krydos will see this...

It's working for me https://stockdiv.com/Stockdiv/Stockdiv

For me too.

[Amoss]

The fact you got online after everything, doesn't say that everything was ok. Please read the following and let me know what can be done. Thanks

 

This was taken in Jun 7th 11:20am (gmt+3). The load averages are again high (4-7). I don't understand the relation between the cpus at the top and in the list. The list shows htop taking most of it with 2% but the totals are 18 and 15.5. Also the LA is high but the high cpu percentage is just htop, I'm sure this is not causing the high LA.

 

Also, what are all the lines I marked with arrows?

 

 

How come vesta is doing so many updates and eventually it's still 0.9.8.24?

 

If the memory is the problem but not because of my stuff then I want to first make sure we rule out everything else. If still we see a problem, then we might go with doubling the memory experiment. I don't want to think everything is ok and then when we decrease it, we'll have problems again. As you can see from the screenshot above and below, it looks like something is still eating my server and it's not my stuff.

 

You said you increased the apache timeout to 60, can you increase it to 5 minutes and also server.xml? what was the timeout before that?

 

After your changes (vesta/timeout) the LA is still high. It took me almost 20 minutes to write this message and all this time the LA was between 4 and 7 and is still is.

 

 

 

Jun 7th or 8th 12:50pm (gmt+3) again a timeout was occurred. Do you see something on your side?

 

Things have calmed down but please take a look at things so we can prevent the next occurrence of it.

---

Here is the mess I was going through for more than an hour yesterday just because I needed to redeploy my war.

 

Jun 9th 1am gmt+3 - I redeployed the war. undeploy took less than a minute, deploying took 7-8 minutes and the LA got to 13 at some point (as you warned me).

Several minutes after deployment was successful and war started working again (getwarversion returned ok) - the LA remain high as 9 causing the other (heavier) requests to fail with 502, I guess due to apache timeout. Here is htop 5 minutes after deployment ended: not sure what is running so much of deployment ended.

 

 

10 minutes after deployment, war only works when asking for its version. Other heavier requests are still failing. and LA jumped back to 10

 

after 30 minutes from redeployment and WAR is not working at all (not even getwarversion), LA is around 10. Starting to think whether to restart tomcat or not, last time that was the only thing that saved the day.

 

40 minutes after redeployment, restarting tomcat , shutdown takes over 10 minutes, at this time nothing works. SSH takes forever to log in. This was taken 50 minutes after starting redeployment, more than 10 minutes after shutdown of tomcat which is still running:

 

50 minutes after, tomcat is still shutting down though La is a bit down to 2.

Just when I wrote the above line, LA jumped to 10

 

After 60 minutes I stopped the tomcat shutdown process and started it again but apparently tomcat was off already but the ssh was stuck. I restarted tomcat. LA is 1-2. War is working. It looks like the tomcat shutdown/restart "fixed" it though I'm sure I'm not suppose to do that every war deployment.

 

I hope that soon you will have time to look at all this and please tell me if this behavior makes sense and if not, based on the above processes and htop screenshots, what can be done about it.

Edited June 9, 2019 by Amoss

[Amoss]

Soon I will need to re-deploy my war and the last thing I want is to go over the previous saga again.

Please let me know if you need more info from me or if nothing is missing, what can be done to avoid it

Thanks

[Amoss]

Lately I started receiving this email when I'm sending an email from my server:

 

This message was created automatically by mail delivery software.
A message that you sent has not yet been delivered to one or more of its
recipients after more than 24 hours on the queue on stockdiv.com.

The message identifier is:     1haCN9-000TIg-Ea
The date of the message is:    Mon, 10 Jun 2019 07:57:13 +0300
The subject of the message is: Re: Feedback

The address to which the message has not yet been delivered is:

    host alt4.gmail-smtp-in.l.google.com [74.125.193.26]

No action is required on your part. Delivery attempts will continue for
some time, and this warning may be repeated at intervals if the message
remains undelivered. Eventually the mail delivery software will give up,
and when that happens, the message will be returned to you.

----

 

The message technically say I should do nothing but because I get a lot of it lately and it means the recipients do not get my email in time and eventually it might not even send the email, please take a look at it. Not sure why it is not being delivered.

 

Thanks

Edited June 11, 2019 by Amoss

[Krydos]

Thank you for making this a post rather than a PM. I always forget to check my PMs, and I've never even attempted to provide support through PMs before you. I'm not ignoring you, I've just been really busy lately with work and real life. You say you're worried about making HelioHost look bad with all of your VPS issues, but there are a couple things here.

 

First, HelioHost has always been completely open and transparent about our strengths AND our weaknesses. Most businesses try to hide everything behind ticket systems where no one can see anyone else's problems. The main problem with that is it increases the workload for the staff because they have to deal with the same problems over and over, where on a forums like we have here anyone can search for their issue and see how other resolved it in the past. Not only does this make things easier for our staff, but it also gives new volunteers a chance to help out, and become promoted to admin themselves. We recently promoted Flazepe for this exact thing. He helped out a lot without asking anything in return, and we gave him a promotion so he can help out even more. Even I was just a regular user looking for customer support a long time ago. I liked the concept that I just outlined, and started helping out myself. Maybe Flazepe will have my job some day. Who knows? Anyways, my point is if someone else has this same sort of VPS issue in the future they can read this thread to see things to try.

 

Second, you're not making HelioHost VPS look bad because you're the only person, out of about 20 right now, that is having these sorts of load issues. It's much more likely it's a software issue than it is any kind of "hardware" issue that would indicate HelioVPS are poor quality. Don't worry about making us look bad by posting publicly.

 

In your screenshots it looks like vesta is still doing updates on its own, which is odd because I unchecked the box for auto-updates, and it appears to still be disabled. You can see it called v-update-sys-rrd.

 

We discussed this before you even bought the VPS. You asked basically, "How will we know if my VPS needs more memory?" This here, exactly what is happening with the load spikes, is what lack of memory looks like. My offer stands: I'll give you double memory for a week for free. If the load spikes get better or go away completely you can make the decision to continue paying for the extra memory after that. I really think this is the best solution.

 

I'm checking on your mail issues now. I'll edit/post again when I have information.

[Krydos]

According to https://forum.vestacp.com/viewtopic.php?t=12995 it looks like your server just needs a reboot to make exim start working right again. Want me to do that for you?

[Amoss]

First of all, thanks for your reply.

 

As you can see from my screenshots, the load issues were when I deployed the new war. And it happened twice in 2 different deployments and it stopped only after shutting down tomcat and restarting it. You say Vesta decided to update itself right at that time, twice? Also, It is still in the same version/build so I'm not sure what is updated so often?

 

What about the differences between the 2 cpus at the top and the cpus in the list? I didn't see any correlation when the high loads happened. I'm asking because I see high loads but I can't know what is causing it.

 

What is my exim version?

 

If the load issues were due to my stuff, I'd go with doubling memory but if the problem is only when/because vesta update itself (are you sure the load issues in my screenshots are due to that?) I first want to rule this out. I searched google for disabling vesta update but found no results. I wonder if my VPS is the only one with this problem.

 

The link you wrote also suggests to delete some files before reboot, I guess I don't really have another option so please reboot but please make sure that my war/site are working afterwards (remember what happened after the last reboot) and let me know after it's done.

 

Thanks

Edited June 12, 2019 by Amoss

[Krydos]

I specifically asked you if the load was because you were undeploying and redeploying and you said the load happened before you did that...