Ashoat

Should have no relation to permissions on zone files, as I didn't end up touching those. However, might have something to do with /var/cpanel/userdata. I fixed the permissions on that folder; is everything working now?

I'm pretty sure this is related to the /var/cpanel/users permissions issue that came after the recent maintenance, except this time it's with /var/cpanel/userdata.

I'm not sure. In the past, reboots have fixed these issues. Can somebody confirm or deny?

I think so. All the files were owned by root's group instead of the relevant user's group. I wrote an awk script to fix it.

I think I've fixed the issue. Is everything working now?

Ermm... there is only 9.7 GiB in total on /var. I'm seeing 1.9 GiB free, which should be enough

It's also worth noting that in addition to adding /var/lib/pgsql I also increased the size of /var/lib/mysql.

Okay, maintenance is complete.

This issue should be corrected now.

Sorry... didn't have time yesterday. Doing it now.

Since we're running out of space on /var for the PostgreSQL databases and mail queues, we're going to create a new partition to store the PostgreSQL databases. This will probably mean up to 12 hours of downtime tomorrow on Stevie. Johnny will be unaffected.

/var/lib/pgsql is only around 1.5 GiB... does it deserve its own partition?

The queue is actually being processed right now, although accounts won't come up until Apache restarts at around 3 AM PST. We don't actually regularly reboot Stevie - servers are supposed to be constantly running

Nice!

No I mean the one guy who keeps using way too much of PgSQL. The process probably spiked server load to the point that the system started thrashing.

This is due to /var being filled up. Mail queues are stored in /var/spool/exim and mailboxes are in /var/spool/mail.

Probably related, but it's also possibly the PgSQL issue. I was unable to SSH in, so I requested a reboot. Currently running a fsck. UPDATE: fsck finished; I'm in. Since we can still get in I'm guessing it was PgSQL and not /var.

Hmm... yeah it looks like PgSQL databases are taking up more space than I expected. Perhaps we should rip a chunk out of sda7 (/home) and use it for /var/lib/pgsql? This will involve a good chunk of downtime.

xaav: It is actually blocked because of that? I thought we've been over it before, and were unable to address the issue.

FYI, I added " > /dev/null 2>&1" to the end so that I don't get emails.

Your account seems to be working... closing.

Looks fine right now. I'm more concerned about /var/lib/mysql...

Ermm... yeah, using that command results in corruption in our account DB. Krydos: do you remember the accounts you ran that command on? If you do, it would be good if you could go through each of them and make sure the account DB has the correct state for them. In the future, use the scripts in ~root/ruby/cli/account.

Looks like this was caused by cPanel's overactive brute-force protection daemon. I've whitelisted Cody on both Stevie and Johnny so this issue shouldn't be occurring any longer.

The NS records don't seem to be pointed at us. http://who.is/whois/superiorapps.com/