After more research this looks like on Dec10, it's as if AWS reverted to a snapshot from Nov8 and continued on. The app logs on the live volume show Entries up to Nov8 and then immediately jump to Dec10 as if all the days in between never existed. Another app using the same volume had the same issue. I'll have to go back but I suspect all the snapshots from Nov8 forward are duplicates until Dec10 at which point history resumed as if Nov8 was the day before Dec10.
Has anybody seen this before? I trolled through trying to see if somehow we did something that somehow restored the main volume from a Nov8 snapshot but nothing in CloudTrail Events show any odd behavior around either Nov8 or Dec10.
After digging around a lot here's what I found:
A snapshot previously used and converted to a volume and attached inherited the same UUID as the live volume. To make a long story short, after some updates and a reboot to the system the system apparently picked the first UUID match for the entry in /etc/fstab. It turns out it mounted the recovery volume that was left attached instead of the proper live volume. Sometime later on a different update/reboot cycle the system was restarted with the recovery volume properly detached and it appeared there was a missing time slice of data.
To prevent this from happening again /etc/fstab has been updated to use devices instead of UUIDs. Now if we have a restore volume attached it won't be in conflict with the live volume. The chances of changing attachment devices in the AWS console are very low so that wouldn't be anywhere near of an issue like UUIDs are.
While AWS didn't indicate this in the support ticket, some of their feedback on logs and information led me to get this one figured out. Thanks to the AWS support crew for this effort.
Taking EC2 Snapshotsasked 4 days ago
Tag Specific Volume in a Multi-Volume Launch Templateasked a year ago
Total EBS Volume for an AWS Accountasked 5 years ago
Getting charged $1 a month for EBS?asked 8 months ago
Is it redundant to have an EC2 instance and its EBS volumes in the same AWS Backup resource assignment?Accepted Answerasked a year ago
How do you find the EBS Volume IDS for a Volume that was created and attached at EC2 Instance Launch Time ?asked 7 months ago
Snapshots (and Volumes created from) Missing changes for a monthasked 2 years ago
creating io2 Block Express encrypted volumes from encrypted snapshots is brokenAccepted Answerasked a year ago
Snapshots of encrypted EBS volumesasked 6 months ago
Snapshot of EBS volume created from snapshotasked 4 years ago