Piotr GRD

Not the nameserver on .131 thing, but maybe it's completely separate problem accidentally started almost at the same time, little bit after Johnny started being inaccessible.

This may be something else than usually. But I may be wrong, of course.

FTP and cPanel is accessible, only HTTP on port 80 is not. cPanel reports practically no server load, but memory usage is surprisingly high (especially swap) as for the server that do almost nothing.

Additionally - which for me is the main factor that it's something different this time, but I may be wrong, of course - the nameserver (located physically on the same server) has stopped to respond for DNS queries on the 64.62.211.131 IP, the one assigned to ns1 subdomain. It does respond on .132, on .133, it does respond on .134, so it works, but does not respond on .131 IP.

@ Geoff

I don't know how about you, but I have just convinced myself that Johnny does affect the whole physical server and nameserver on it. Look at the time when Johnny did serve only the "DIVA Development", majority of requests ends up with 404 responce, no any PHP were executed and... nameserver responces were definitelly more stable.

Image: http://i52.tinypic.com/141lf06.png

All accounts located on server Johnny display the same at the moment.

At the moment all accounts on the server Johnny seems to contain exactly the same files of some "DIVA Development". Try smashleaks.heliohost.org/website%20home%20page.htm or johnny.heliohost.org/website%20home%20page.htm or any other URL that points to the Johnny. If you still see some of your original pages (ie. the main one) then it's just the cache of your browser most likely.

I don't think that it's just you.

At the time of your post all (or many) PHP scripts on Johnny did return 500 (pure HTML pages worked fine).

Later the websites on Johnny worked fine and your "decoded.heliohost.org" did return "account queued" page.

And now all the Johnny accounts seems to contain the same "DIVA Development" website files.

So...

CloudFlare is not that great for dynamic pages in my opinion.

It works in the way that you are pointing your domains/subdomains to their servers with the proper CNAME records.

The visitors are sending requests for your pages to their (multiple) servers. Then their servers are requesting the page from original source and send it to the visitor. In other words - it works as a reverse proxy.

While it works fine for static pages as these may be cached on their servers, save the bandwidth of your original server and even if it is down the page will be displayed to the client from the copy, but it's not that great for dynamic pages, as the "proxy" server needs to access the original server everytime, anyway.

I did see the pages that use CloudFlare already and in a case of original server problems for dynamic pages I've only see a CloudFlare message about problem after many seconds of waiting time instead of request timing out like it is normally.

Some other hosting recently did turn it ON and integrate in cPanel for all the clients, I don't know if it works as espected for them. Dynamic pages that uses it are not faster if the original server is slow, at least that's my personal experience.

Other people may have different opinion, definitelly it have to have some advantages if so many people are using it already.

Stevie is intermittently slow with the responces, so this may be the reason of your problems, too.

More in here: http://www.helionet.org/index/index.php?showtopic=9276

@ Geoff

I don't know how the physical server resources are shared between the two VPS'es, but I would guess that one could affect the other. Even Charlie's responce times (not displayed on my monitor, but I log them) are vary, not that much but sometimes when Cody and Johnny are slow, Charlie's responce time is longer, too.

Yes, HelioHost nameservers are intermittently inaccessible.

Apart of recent longer problem with Cody (mentioned in the news) that has been resolved ~24 hours ago, in the "normal" day the nameservers are quite often inaccessible. You can check this ie. on my server monitor (link in my signature) - browse the June 6th and before to see the average condition.

Some solution may be using some other nameservers (external, ie. from your domain registrar, valid A/CNAME/MX etc. records needed to be configure in there, of course) for your domains hosted in here, in such case HelioHost nameservers failures will not affect accessibility of your websites hosted in here.

@ users

If you would use external (not HelioHost) nameservers for domain names hosted in here, your websites either on Stevie or on Johnny would be still accessible in such case like this one. It's just were the problems with Cody - which handle this forums, hosting main page and nameservers, and only because of not functional nameservers users websites were unavailable.

@ djbob

I do know that having two seperate nameservers while there is a lot of domains/subdomains is a pain and takes a lot of servers resources to share/copy the zones between, but obviously having two nameservers would save users websites from being unavailable in such situation. Considering that Johnny makes the whole Charlie (including nameserver) intermittently unavailable even in normal day, this could be good idea to have two nameservers (of course depends on how much additional stress on servers would cause zone transfers between the two).

Side note - that was really funny to see server load 0 (zero) on Stevie and almost 0 on Johnny (at the time when I've checked). ; D

@ witchcraft

You would have to ask yourself:

- why someone/some account didn't abuse the server so much before the power failure and started to do this instantly just after power failure?

- how is this possible that the "abuse process" slows down the server while not generating any significant load on CPU or hard disk drive? (CPU usage and disk I/O are two factors considered while counting the "server load", which is low most of the time as mentioned by jje)

Personally I think that it is some hard to find hardware problem. It would be "better" if the selected part of the hardware would fail completely so it would be more easy to find and replace.

If it works - keep it. Does it matter if this will be iframe, image or any other *external* file loaded? With AJAX requests you would also have to warn every user that javascript *needs* to be turned on in the browser. Some people may also have browsers set (which is not that rare! - this is setting for security reasons) to not execute javascripts from different domain than they are on so while being on ie. example.com:2082/frontend/x3/index.phpcp javascript/AJAX request from heliohost.org may be not loaded for them (and again, it's usually silent error in the browser).

If this would be process included in the cPanel - then it's different story.

I may be wrong, but my theory is that some people's accounts are suspended because the script does not load for them, even if they're login to cPanel. Look how often heliohost main page and forums times out for you or loads very slowly. It's intermittent but one may have "bad luck" and while being logged in in cPanel the mentioned script actually does not load. And either if it will be iframe, image or whatever - it will be silent error in the browser, so user is not aware that this time he/she logged in the cPanel for nothing, it was useless and will not prolong the account.

While being on a subject...

(...) If you'll look at the source code at your cpanel, you'll see inside of an iframe, the renew script with your username. (...)

So do I understand you correctly?

You actually need to access the source of that iframe and not the login process to the cPanel itself?

This issue has been mentioned already by rvt 2 months ago:

http://www.helionet.org/index/index.php?s=...ost&p=61236

However his solution was to change redirection instead of link in email.

I'm not sure if it was fixed and then reverted back (if still occur) or something else happend and cause further issues.

I have mentioned this correlation between server responce time and websites being unavailable 16 days ago in here while thinking that djbob did something at that time, but that was unfortunatelly the same kind of failure like it is now. Anyway - for someone who knows the server configuration and knows how it works when websites does not work etc. this might be some tip as to the reason of this weird slowness.

Some of this could be time spent on looking up NS records. (Which isn't even on stevie)

In a case of my monitoring server - I send the requests directly to the server with using IP address, so any DNS issues does not affect my results. What's more - I see that majority (if not all) "A" records on heliohost nameservers have TTL set to 4 hours, so in such case every person would have one slow query per 4 hours only for selected domain name, later DNS entries are cached on his/her computer. Additionally - as you may see just now for hours - while Johnny does not work, Cody is very fast, so the responces from nameserver, and situation on Stevie does not change.

It looks like that indeed it may be related to some hard drive problem.

Since more than 3 hours the waiting time for the response from the server is fast with no delays. The websites on it are missing, though, so it may suggest that the hard disk drive has been unmounted (or whatever else technically) for the time of the corrective fsck mentioned by djbob - if this is what just has been started by him. If not, then anyway this shows the possible correlation.

By the way - thank you for using my HelioHost Server Monitor, I'm happy if it's at least a little bit useful. : )

@ sosunlight

The server "Stevie" is currently down, that's why you can not access it.

You can keep an eye on the accessibility of the servers by looking at this server monitor (for guidance only):

http://grd.net.pl/heliohost/monitor/

Informations about current problem you can find ie. here:

http://www.helionet.org/index/index.php?s=...ost&p=63863

What to do for being moved is described here:

http://wiki.helionet.org/wiki/Moving_your_account

But note that this page is currently not available (I think it's hosted on Stevie, but I may be wrong, didn't check enough).

Keep in mind, however, that Stevie is described as being more stable, while Johnny is described as having potentially much more problems which seems to be true by looking on my linked above server monitor archive results for the last couple of days (before current downtime).

(...) We will need a few more days to verify that johnny is completely safe.

I know you can login into cPanel, anyway, but I'm hope you're also aware that johnny.heliohost.org:2082 page does not contain login fields. Just saying for a case you missed this.

(...) If anyone knows how to truly prevent browsers from caching, I'd love to hear it.

I'm not sure what is the "cache problem" in here, but I use something like this:

header('Expires: Mon, 01 Jan 2001 00:00:00 GMT'); // date in the past
header('Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, max-age=0');
header('Pragma: no-cache');

Of course if in such cases like above mentioned redirection takes a place - such headers have to be sent with the redirection "page", not with the landing page where visitors are redirected to. If redirection is done with .htaccess or with some other method (not with PHP script) - I don't know how to attach these headers, unfortunatelly.

Quote: "You are user number 10 of 50 allowed."

Good that simple restart did help.

IF. If this is the bug that I'm relating to, then it appears usually when server is slow and/or some problems (even temporary, for a milisecond) appear exactly when someone is connecting and/or disconnecting. It's quite rare situation, even very rare, but obviously as Stevie is quite often slow - such bug (if it's the case in here) may appear more often than somewhere else.

It's related to Pure-FTPd, as I didn't notice such situation on servers with different FTP software installed and I do monitor dosens of servers. I don't know exactly how it works and why the bug happend, however as the end user for over a year on many servers trying to connect every 2 minutes 24/7 I have such experience. It also may affect a single user on the limit of maximum allowed simultaneous connections per IP.

A friend of mine did use fsockopen, so could get all the data sent by the server to the client. Obviously all the FTP commands have to be specified correctly and sent to the server, too, using this method, so you need to know enough this protocol.

A little more simple to use is cURL, but in such situation it's return only "This doesn't seem like a nice ftp-server response" for me, not the real server responce.

For those who interested - my monitoring shows orange (no valid responce) for FTP in such case. Red for time-out or offline.

http://grd.net.pl/heliohost/monitor/

edit:

By the way - did you try to reset / turn off and on again the FTP server?

Because I know of existence not that much known bug in Pure-FTPd that can cause such situation. It makes that really already unexisting connections are still considered as existing and counted to the limit. So there can be reported 50, while there is really no 50 of them.

But it may be not the case in here, of course.

edit2:

I dunno, 42k users stay on ftp on the whole day? ...

"421" is the FTP reply code. Error message is "50 users ...". It's not 42150 open connections, just 50.

What I'm pointing is that that on old - Stevie - nameserver is still old A record for heliohost.org, on new - Charlie - nameserver is new A record already. And couple of hours ago from some location - like ie. Germany, Netherlands, California (USA) (such locations I did check) - still old (Stevie) nameserver has been queried while asking ns1/ns2.heliohost.org for the records for heliohost.org and www subdomain of it.

But that's a small thing, of course, just couple of hours. ; )

(...) while DNS records of type "A" usually have a TTL of about 4 hours. (...)

I would say that it has usually such TTL that admin wants it. ; )

Ie. I use 1 hour, but if I have plan to change IP, I try to set it temporarily for 5 minutes or so, for propagation being as short as possible.

Small tip for future such steps: you could make the DNS propagation delay for the whole world much shorter with setting new IP for A records for heliohost.org on old nameserver(s) (on Stevie), too. At the moment it's too late, though, everyone's systems should use new (Charlie) ns1/ns2.heliohost.org. But couple of hours ago from some locations still old nameservers has been queried. Without changing entries on old nameservers it takes ~24 hours for everyone to access new location (if I'm right), with changing entries on old nameservers (or switching location of nameservers 1 more day before) it could take only 4 hours (with current TTLs). Just a small thing, couple of hours difference doesn't mean much. ; )

When you'll move helionet forums, though, to avoid some people posting on old, some on new location, you can set TTL for being shorter than 4 hours. At least 4 hours before changing IP itself for A record, of course, for such step having any effect. After changing IP TTL can be set for being longer again.

Forgive me if you're aware of all these things and you just don't bother much for the DNS propagation being couple of hours faster or slower.