AWS Bahrain (me-south-1) Outage – EBS Snapshot Issue After Recent Incident

Hi,

At this point, you may want to take a look at this article to see if it can help you in any way, though even that approach is likely very much hit and miss right now. If the instances (and their respective volumes) are still reachable you may need to resort to manual copies over the network which again may or may not work right now.

For future reference even if that doesn't help you right now, you should never have a production disaster recovery approach that is reliant on the region where the production workload is running.

I understand you're facing a critical situation with snapshot creation failures in the me-south-1 region. While I cannot comment on specific regional incidents or outages, I can provide some guidance on troubleshooting snapshot issues and potential alternatives.

When snapshot creation fails, there are several common causes to investigate:

Common Snapshot Issues:

• IAM Permissions: Ensure your IAM policies include the necessary permissions: ec2:CreateSnapshot, ec2:DeleteSnapshot, ec2:DescribeVolumes, and ec2:DescribeSnapshots. If your volumes are encrypted, you'll also need appropriate AWS KMS permissions.
• Snapshot Quota: Check if you've reached the maximum number of concurrent snapshots allowed in your account.
• Volume Availability: Verify that the volumes are in an available state and haven't been deleted or moved.
• Throttling: If you're attempting multiple snapshots simultaneously, you may be hitting service quotas for concurrent snapshot operations. Try staggering your backup attempts to reduce concurrency.

Alternative Data Recovery Approach: If standard snapshot functionality remains unavailable, there is an alternative method using the EBS Direct APIs and tools like coldsnap. This approach allows you to:

1. Download snapshot data block-by-block from existing snapshots (if you have any)
2. Transfer the data to another region
3. Recreate volumes from that data

However, this method requires existing snapshots to work from and sufficient local disk space for the transfer operation.

Immediate Recommendations:

• Monitor AWS Service Health Dashboard for any regional service updates
• Attempt snapshot creation during off-peak hours to avoid potential throttling
• If you have existing snapshots, consider using the EBS Direct APIs approach as a backup strategy
• Contact AWS Support directly for urgent production issues, as they can provide real-time assistance and region-specific guidance

For production-critical situations like yours, I strongly recommend opening a support case with AWS Support for immediate assistance and region-specific updates.
Sources
Troubleshoot EBS volume backups in AWS Backup | AWS re:Post
How do I copy an EBS Snapshot to another region using coldsnap and the EBS Direct APIs? | AWS re:Post