[Solved] VPS filesystem has become read-only

[badrihippo]

Hi all...my VPS filesystem has suddenly become readonly

 

I suspect it happened after this week's reboot, because I'd checked the read-only parts but never explicitly done anything requiring writing till now.

 

I see this related error message in dmesg (let me know if you want the full output):

[154995.476735] EXT4-fs error (device sda1): ext4_journal_check_start:61: Detected aborted journal
[154995.478036] EXT4-fs (sda1): Remounting filesystem read-only

Any idea how to fix this (redoing mount doesn't work and fsck doesn't seem to be installed)? Would it be possible to do a manual reboot from your end?

 

My server is hippo2.heliohost.org, running Debian 10. (I don't have any irreplaceable data left, so a full reinstall is also an option, although not ideal).

[wolstech]

This one's a Krydos question...

[Krydos]

I tried logging in with the default password that I sent you, but it said it was invalid. Did you change the password? If so I'll need the new password to do anything. Obviously don't post it on this forum, but you can email it to me if you want.

 

The other option is I can just yank the power cord out of the wall and then power it back on. That's unlikely to do anything other than maybe cause more filesystem corruption.

 

I think this was actually all my fault in the first place now that I think about it. I was kind of in a rush to get the VPS done and I forgot to install vmware tools. That package allows my to cleanly shutdown or reboot a VPS through my admin control panel. Without vmware tools when I needed to power your VPS off for the maintenance I just had to yank the power on it, which is probably what caused the corruption in the first place. Sorry about that.

[badrihippo]

Oh okay, that makes sense. No worries, but could you perhaps install vmware tools now, once you're done, to prevent it from happening again? I don't mind a a few days' wait.

 

I've sent you the new user and root passwords by email.

[Krydos]

Alright, it looks like you're all good to go.

 

One thing I noticed is /usr/sbin/ isn't on the path. So when you said fsck isn't installed, it actually is at /usr/sbin/fsck, but running the command fsck can't find it. The command /usr/sbin/reboot isn't on the path either, so issuing the command reboot won't work either. Something to look in to, or keep in mind for the future when commands don't work as expected.

 

I was able to log in and fsck your hard drive. It looks like it fixed all the errors. Then I was able to cleanly reboot your VPS. The reboot took about 30 minutes though because one of your processes really didn't want to die, but I just waited it out and eventually it shut down. Not sure if all reboots will take that long, or that process was just having a bad day. I installed open-vm-tools package so if we need to shutdown or reboot your VPS in the future for maintenance this won't happen again. That doesn't happen very often though.

[badrihippo]

Great, thanks a lot! Now that you mention it, I did notice /usr/sbin/ being off the path at some earlier point but forgot about it when this new problem came up. That was a bit silly of me. I'll keep it in mind next time.