How to Quickly Restore MongoDB Data for an Impaired Volume?

Question

I have 3 EC2 instances mongo01, mongo02 and mongo03 running MongoDB 4.2.4 with replication. After yesterday's AWS data center power outage, mongo02 cannot start due to an impaired volume with the MongoDB data.

I am wondering if the following steps can let me bring mongo02 back online quickly:

1. create a snapshot of /data in mongo01
2. create a new volume 100Gb with the snapshot in (1)
3. attach the volume in (2) to mongo02, set device name /dev/sdf
4. start mong02
5. fix /etc/fstab (ie change UUID)
6. mount /data

Answer

The mentioned steps should work for getting 'mongo02' back up. In case you need a walkthrough with an AWS Technical Support Engineer, feel free to open up a case with AWS Premium Support and choose method as either Chat/Call. You can also link this re:Post question for their reference in your case notes.

Going forward, I suggest looking into Amazon Data Lifecycle Manager. You can use Amazon Data Lifecycle Manager to automate the creation, retention, and deletion of EBS snapshots and EBS-backed AMIs. Please find more information here:
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html

The following procedure shows you how to use Amazon Data Lifecycle Manager to automate Amazon EBS snapshot lifecycles:
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-ami-policy.html

For context, if we had a known working snapshot for EBS volume associated with 'mongo02', we could have just created a volume from snapshot and replaced it. We would not have needed to edit `/etc/fstab` or any other configurations. Since this thread is about quick recovery and having the lowest Recovery Point Objective and Recovery Time Objective, I would suggest looking into automating snapshot lifecycles.

How to Quickly Restore MongoDB Data for an Impaired Volume?

Relevant content