Suggestion: Reduce Server Ftp Load By Supporting And Encouraging Use Of Rsync Protocol

[kiwiphnx]

Hello Helionet,

 

My observations of using Nautilus' DAVS support to transfer files to/from Tommy (I can't seem to get SFTP/FTPS to work reliably for some reason) have been that file transfers seem to take a lot longer than they should. It also occurred to me that the majority of FTP-like traffic would be clients either modifying and re-uploading their sites, or taking backups. Therefore, it seems there is the potential for a performance gain by supporting, and encouraging the usage of, a differential file transfer protocol such as RSync; possibly transported over SSH or SSL to provide encryption.

 

My suggestion is to have a look into this (there is a Node.js RSync implementation that could be used to provide cross-platform support via a web browser) and see what potential performance gains, if any, can be had over vanilla SFTP/FTPS/DAVS; although I suspect the SSL encryption cost would far outweigh rsync's file checksumming cost, especially if the results of the latter can be cached.

 

[Krydos]

The fastest way to upload files is to upload an archived tarball of the directory structure and files, and then extract it on the server. The slowness of FTP/SFTP/WebDAV/etc. is probably because of the number of files not the transfer rates.

 

I just tested the upload/download speed of Tommy and he's right around 100 Mbit/s both directions. Is your home internet connection upload speed higher than that? Do you consider 100 Mbit/s slow?

 

Also, a lot of developer tools like Visual Studio Code, Dreamweaver etc. that I've seen support differential file transfer anyways over regular (S)FTP.

 

I would use rsync though if it were available, but as hard as FTP is for most people to figure out I doubt many would use it. I'm actually a little surprised that cpanel doesn't support it now that I think about it.

[kiwiphnx]

Thanks for the response and the feedback. No, I don't consider 100Mbps slow; in fact, I'd be very happy to obtain a transfer rate that was anywhere near the 1Mbps that our DSL connection would allow (for uploads).

Perhaps it is our round-trip-time to the server that is the problem; I'll try your suggestion of tarballing and see if it helps our specific situation.

 

Another, possibly related phenomenon is that we often experience lag times of up to 3 seconds when accessing our website over HTTPS. I assumed this was due to server load, as we don't seem to have such problems with other NZ-to-US traffic.

[Krydos]

You're on Tommy which offers instant domain activations. What this means is apache has to restart. A lot. You're on a server with thousands of other people and each time someone adds or changes a domain apache restarts, and each time apache restarts there is a lag of a few seconds where pages won't load for a bit. This is one of our main features of the server. We could reduce the restarts like Johnny or Ricky, but then domains would take hours or days to start working. We keep a close eye on the response time of Tommy though as he's our flagship server. For the last 4 hours his median response time has been .010 seconds, and his mean response time has been .143 seconds. We think that's an acceptable balance.

 

If you want a server that only restarts apache for you and you alone you need to be the only person on the server. Check out https://www.heliohost.org/partners/vps for that service.

[kiwiphnx]

That's actually a very good explanation of the highly variable response times of Tommy; and I apologise if I came across as self-centered (that wasn't the intention). I believe I want the same thing you do; for HelioHost to be the best possible service it can be; and I am willing to work with you to help achieve that goal. That said, I think the service you offer already is pretty terrific and your passion towards the community is admirable. I represent and can find clients that would be prepared to back you with real money if we can find a way to make your services more dependable. Please help me to find a way to help you achieve this goal.

 

One suggestion I could make (apologies if you've already considered this) is to put a reverse-proxy (e.g. Squid or Apache with sub-requests) in front of your webservers (physical or processes), configured such that it can immediately serve a cached response (appropriately marked with a 502 or 503 response code if the cache is stale) while the origin webserver is restarting. Doing so might even help reduce the load on your origin servers for static content such as images, giving Apache more time to generate dynamic content.

[Krydos]

That's a good idea. We'll keep that in mind as we continue growing and buying more servers.

 

In the meantime you can use a free cloudflare account to serve static content while the server is slow and cloudflare also helps reduce server load as well as speeds up response times to most everyone worldwide.

[kiwiphnx]

Thanks for the suggestion. You'll be happy to know we already use Cloudflare to cache and distribute our static content (which helps immensely). Unfortunately, I am yet to find a reasonable solution to the problem of caching WordPress' output. I have actually gone to the point of installing various caching plugins (specifically Autoptimize and Cache Enabler, although I have tried many others along the way) and configuring Apache to immediately serve pages from the cache (bypassing WordPress completely) whenever possible.

Unfortunately, WordPress relies on cookies to determine whether a user is logged in, which necessitates the sending of a "Vary: Cookie" header, which effectively kills any shared caches along the way.

 

If you go with a reverse proxy, it may also be a reasonable compromise (security-wise) to offload SSL from the origins to the proxy, freeing up more valuable CPU resources. If required, local traffic between the origins and the proxy could still be encrypted and sent over an aggregated, reusable connection, which would at the very least avoid the expense of repeatedly setting up and tearing down SSL connections at the origin.