Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Best Practices: Create Backups Before Making Changes to Production EC2 Instances

5 minute read
Content level: Foundational
2

This article outlines best practices for protecting your production workloads and introduces AWS services that automate backup management.

Overview

Many AWS customers experience unrecoverable issues such as kernel panics, GRUB boot failures, or broken configurations after applying patches, OS upgrades, or configuration changes to production EC2 instances without a backup in place. These situations often result in extended downtime and data loss that could have been avoided with a simple EBS snapshot taken before the change.

This article outlines best practices for protecting your production workloads and introduces AWS services that automate backup management.

The Problem Common scenarios where lack of a pre-change backup leads to extended outages:

  • Kernel panic after patching – A kernel update renders the instance unable to mount its root filesystem. Without a snapshot, there is no clean state to restore from.
  • GRUB boot loader failure – An OS upgrade or manual GRUB configuration change leaves the instance stuck at the GRUB rescue prompt.
  • Broken application or system configuration – A change to /etc/fstab, networking, or security settings prevents the instance from booting or accepting connections.

In each of these cases, recovery without a backup requires manual intervention through rescue instances, which is time-consuming, error-prone, and not always successful.

Best Practice: Always Create a Backup Snapshot Before You Make a Configuration Update/Change

Rule of thumb!! Never apply a change to a production server without creating an EBS snapshot of every attached volume first.

Manual snapshot workflow

Before performing any change (patching, upgrading, configuration modification):

  1. Identify all attached EBS volumes for the instance.

  2. Create snapshots of each volume: aws ec2 create-snapshot --volume-id vol-0123456789abcdef0 --description "Pre-patch backup 2026-06-24"

  3. Wait for snapshot completion before proceeding with the change.

  4. Verify the change works correctly.

  5. Retain the snapshot for a defined rollback window (e.g., 7 days) before deleting.

Rollback procedure If the change causes a failure:

  1. Stop the affected instance.
  2. Detach the failed volume.
  3. Create a new volume from the pre-change snapshot.
  4. Attach the restored volume and start the instance.

Automate Backups with AWS Services

Manual snapshots are effective but rely on human discipline. AWS provides two services to automate this process.

Amazon Data Lifecycle Manager (DLM)

DLM automates the creation, retention, and deletion of EBS snapshots on a schedule.

Use DLM when you want to:

  • Automatically snapshot EBS volumes on a recurring schedule (e.g., daily, hourly)
  • Enforce retention policies (keep the last N snapshots or retain for N days)
  • Tag-based targeting (snapshot all volumes with a specific tag)

Getting started:

  1. Open the EC2 console → Lifecycle Manager → Create lifecycle policy.
  2. Select "EBS snapshot policy."
  3. Define target volumes by tag (e.g., Backup=true).
  4. Set the schedule (e.g., daily at 02:00 UTC).
  5. Set retention (e.g., retain 7 snapshots).

Documentation: https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-lifecycle.html

AWS Backup

AWS Backup provides a centralized, fully managed backup service that works across multiple AWS services (EC2, EBS, RDS, EFS, DynamoDB, and more).

Use AWS Backup when you want to:

  • Centralize backup management across multiple AWS services
  • Define backup plans with rules for frequency, retention, and lifecycle (transition to cold storage)
  • Enforce backup compliance with AWS Organizations policies
  • Enable cross-region and cross-account backup copies for disaster recovery
  • Use backup vaults with access policies for additional security

Getting started:

  1. Open the AWS Backup console → Create backup plan.
  2. Choose "Build a new plan" or start from a template.
  3. Define backup rules (frequency, retention, destination vault).
  4. Assign resources by tags or resource IDs.
  5. Optionally enable continuous backup for point-in-time recovery.

Documentation: https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html

Comparison: DLM vs. AWS Backup

CapabilityDLMAWS Backup
EBS snapshot automation✓✓
Multi-service backup (RDS, EFS, etc.)✗✓
Centralized management console✗✓
Cross-region copy✓✓
Cross-account copy✓✓
Compliance and audit reportingLimited✓
CostNo additional chargePer-GB storage charges
Point-in-time recovery✗✓ (for supported services)

Recommendation: Use AWS Backup for comprehensive, centralized backup management. Use DLM if your needs are limited to simple EBS snapshot scheduling.

Additional Recommendations

  1. Test your backups – Periodically restore from a snapshot to verify the backup is usable.
  2. Tag your resources – Use consistent tags (e.g., Environment=Production, Backup=Required) to ensure all production volumes are included in automated backup policies.
  3. Document your rollback plan – Before every change, write down the specific steps to revert if something goes wrong.
  4. Use maintenance windows – Apply changes during low-traffic periods to reduce blast radius.
  5. Consider creating an AMI – For major OS upgrades, create a full AMI (which includes all volumes) in addition to individual snapshots.
  6. Enable snapshot lock – For critical backups, use EBS Snapshot Lock to prevent accidental or malicious deletion during the retention period.

Summary

StepAction
Before any changeCreate EBS snapshots of all attached volumes
For ongoing protectionConfigure DLM or AWS Backup policies
After a successful changeRetain snapshots for your defined rollback window
If the change failsRestore volumes from the pre-change snapshot

The cost of a snapshot is negligible compared to the cost of production downtime. Make pre-change backups a non-negotiable part of your operational procedures.

Related Resources

No comments

Relevant content