EC2 instances with root FS read only after reboot

0

Hi,
Since last week I've got 2 instances (i-0a900e278e3bdd7e6 & i-0fae9341438917bee) where root filesystem went read only after a reboot

  • the two instances use ubuntu 16.04
  • after boot, I can remount / rw
  • fsck offline with volume mounted on another instance gave no error
  • dmesg don't give something like "remounting / read-only"
  • first systemd service to failed at boot
    [FAILED] Failed to activate swap /var/swap.1.
    See 'systemctl status var-swap.1.swap' for details.
  • here's the ouput of systemctl status var-swap.1.swap
● var-swap.1.swap - /var/swap.1
Loaded: loaded (/etc/fstab; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2018-06-18 18:13:39 CEST; 19h ago
What: /var/swap.1
Docs: man:fstab(5)
man:systemd-fstab-generator(8)
Process: 443 ExecActivate=/sbin/swapon /var/swap.1 (code=exited, status=255)
Jun 18 18:13:39 HOSTNAME systemd[1]: Activating swap /var/swap.1...
Jun 18 18:13:39 HOSTNAME swapon[443]: swapon: /var/swap.1: swapon failed: Read-only file system
Jun 18 18:13:39 HOSTNAME systemd[1]: var-swap.1.swap: Swap process exited, code=exited status=255
Jun 18 18:13:39 HOSTNAME systemd[1]: Failed to activate swap /var/swap.1.
Jun 18 18:13:39 HOSTNAME systemd[1]: var-swap.1.swap: Unit entered failed state.

Is there something else I can check ? Do I need to destroy and recreate?

EDIT1: Later, i'll try to create an AMI and run from it.

Thanks,

Edited by: Lionel-CleverAge on Jun 19, 2018 6:13 AM

asked 6 years ago1794 views
1 Answer
0

Hey Lionel!

Just thought I would update this thread with the solution. I managed to reproduce your issue by launching a fresh instance from the same AMI as yours ami-05851072 and adding the following mount instruction in /etc/fstab :

$ echo "/dev/xvdf1  /tmp  ext4  rw  0  2" >> /etc/fstab  
$ reboot  

The commands above is supposed to mount /dev/xvdf1 on the directory "/tmp" when the system boots. My instance managed to boot with the root partition mounted as Read-Only just as you experienced. Turns out there is a bug in cloud images of RedHat 7.0 that causes the root partition to be mounted as Read-Only when a device is mounted on the directory /tmp. The most annoying thing about this bug is that since it causes the root filesystem to be mounted as Read-Only, the system will not able to log any details we could use to identify this issue. This explains why it's hard to diagnose this issue from the system log files. Anyways, it seems this was resolved in RedHat 7.2 but you can use the following command to fix this bug on your system:

$ sudo systemctl unmask tmp.mount && sudo systemctl enable tmp.mount

Verify that tmp.mount unit is enabled by running this command:

$ sudo systemctl list-unit-files tmp.mount
	
	UNIT FILE STATE
	tmp.mount enabled		<===== as you can see "tmp.mount" is now enabled.

Reboot your system to see if this resolved the issue:

$ sudo reboot

The command above allows /tmp on tmpfs to be mounted by default. Mounting of /tmp on tmpfs is disabled by default on ALL cloud images to discourage people from bind mounting tmp which is considered bad practice in cloud environments. However, there seems to be a bug in systemd on RedHat 7.0 that causes the root filesystem to be mounted as RO if tmp.mount is disabled AND there is a mount instruction for /tmp in /etc/fstab. You can follow this bug on this site [1] for more information about it.

References
[1] Red Hat Bugzilla – Bug 1243494 - https://bugzilla.redhat.com/show_bug.cgi?id=1243494

profile pictureAWS
answered 5 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions