Issues Mounting an Encrypted EBS Volume After Crash

Question

Running patched Ubuntu 20.04 server on AWS to host NFS volumes (from EBS) encrypted with Cryptsetup and formatted with XFS.

We have had this scenario happen a few isolated times across several different servers in different regions with the same setup. These servers have been running without issues for many years with no changes.   Drives are defined properly in /etc/fstab and /etc/crypttab.

What seems to happen is...

* NFS services appear to hang/crash
* On the NFS server itself, the volume is still fully mounted and accessible, but other clients cannot reach it via NFS
* any restart of the NFS services hangs as well and there is no logging of an issue in syslog or anywhere else about why
* running the cryptsetup luksOpen works without issue, but a mount command simply hangs with no response or logging. cryptsetup repair shows no issues.
* restarting or stop/starting the server, has the server return, BUT the NFS volume is not mounted. The step above has the same results.
* detaching and reattaching the volume changes nothing either
* xfs_repair does not reveal any issues

However, if I detach the volume, attach it to a new, fresh Ubuntu install instance, and run "cryptsetup luksOpen" on the device successfully...it then mounts!  And...if I then detach the volume, attach it BACK to the original server, the volume will mount just fine.

Have any others seen this and used another solution to be able to mount a disk?  I do not believe that the new server instance is doing any sort of repair on the volume.

re:Post Agent · Answer

This issue appears to be a complex interaction between encrypted EBS volumes, XFS filesystem, and NFS services. While the exact cause is not immediately clear, there are several potential factors and troubleshooting steps to consider:

1. Filesystem corruption: Although xfs_repair doesn't reveal issues, there might be subtle corruption that's not easily detectable. The fact that attaching to a new instance allows mounting suggests this might be the case.

2. Stale mount state: The original server might have a stale mount state that prevents remounting. This could explain why attaching to a new instance works.

3. Cryptsetup issues: While cryptsetup luksOpen works, there might be underlying issues with the encrypted volume that are resolved when attached to a new instance.

4. NFS service conflicts: The hanging NFS services might be interfering with the mount process.

To troubleshoot and potentially resolve the issue:

1. Before detaching the volume, try using the 'mount -o nouuid' option. This can sometimes resolve XFS mount issues.

2. Check system logs thoroughly for any cryptsetup or mount-related errors.

3. Ensure that all NFS-related processes are fully stopped before attempting to mount.

4. Try mounting with different mount options, such as 'noatime' or 'ro' (read-only) to see if it affects the outcome.

5. Consider running a more thorough filesystem check using xfs_repair with advanced options.

6. If possible, create a snapshot of the volume before troubleshooting to ensure data safety.

If these steps don't resolve the issue, the current workaround of attaching to a new instance first seems to be effective, albeit inconvenient. This suggests that the new instance is indeed doing something to "reset" the volume state, even if it's not an obvious repair operation.

For a long-term solution, consider implementing regular filesystem checks, monitoring for early signs of NFS issues, and having a documented recovery process that includes the working workaround of attaching to a new instance first.  
**Sources**  
[Retrieving Data and Files from crashed EBS Volume (XFS) | AWS re:Post](https://repost.aws/questions/QUKzrSaLEkRH28MJejKTRZ1A/retrieving-data-and-files-from-crashed-ebs-volume-xfs)  
[EC2 Instance Linux data disk mounting lost after stop and start of instance | AWS re:Post](https://repost.aws/questions/QUQ3Ct6xUySx2ldBgFYhMijA/ec2-instance-linux-data-disk-mounting-lost-after-stop-and-start-of-instance)

Issues Mounting an Encrypted EBS Volume After Crash

Add your answer

Relevant content