Skip to content

Why Multi-Region Replication Does Not Replace Backups — Protecting Databases and Streaming Analytics Pipelines from Ransomware

8 minute read
Content level: Advanced
0

A common misconception in multi-region architectures: "I have Aurora Global Database replicating to a secondary region — I'm protected from data loss." This is false for one critical threat class: data corruption that propagates through replication. Ransomware, accidental bulk deletes, application-level logic bugs, and malicious insiders all generate valid write operations that replicate faithfully to your secondary region.

How Replication Propagates Corruption

Consider this architecture:

┌─────────────────────┐         Replication (< 1s)        ┌─────────────────────┐
│  US-WEST-2 (Primary) │ ─────────────────────────────────→ │ US-EAST-2 (Secondary)│
│                     │                                    │                     │
│  Aurora Writer      │                                    │  Aurora Reader      │
│  (compromised)      │                                    │  (also corrupted)   │
└─────────────────────┘                                    └─────────────────────┘

Attack timeline:

TimeEvent
T+0sAttacker encrypts/corrupts primary database
T+<1sAurora Global Database replicates corrupted writes to secondary
T+5sBoth regions contain encrypted/corrupted data
T+60sTeam discovers the issue
T+61sNo clean copy exists anywhere in the replication topology

This applies equally to:

  • Aurora Global Database (async replication, typically <1s lag)
  • DynamoDB Global Tables (multi-region, multi-active)
  • S3 Cross-Region Replication (objects replicate including overwrites/deletes)
  • EBS snapshots copied cross-region (if source is already corrupted)
  • FSx replication / SnapMirror (mirrors corruption faithfully)
  • Kinesis Data Streams cross-region replication (corrupted records flow downstream to analytics consumers)

What Replication Protects Against vs. What It Doesn't

ThreatMulti-Region ReplicationIndependent Backups
Regional infrastructure failure
AZ impairment
Hardware failure
Ransomware / encryption attack
Malicious insider (DELETE FROM)
Application bug writing corrupt data
Accidental DROP DATABASE
Compromised admin credentials✅ (if isolated)

The Solution: Defense-in-Depth Backup Architecture

Layer 1: Point-in-Time Recovery (PITR)

Enable PITR on all critical databases — this gives you continuous, granular recovery independent of replication:

# Aurora — PITR enabled by default, 1–35 day retention
aws rds modify-db-cluster \
  --db-cluster-identifier my-cluster \
  --backup-retention-period 35

# DynamoDB — enable PITR
aws dynamodb update-continuous-backups \
  --table-name my-table \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true

Why this helps: PITR maintains a transaction log independent of replication. You can restore to any second before the corruption event, even after replication has propagated the damage.

Layer 2: Cross-Account Backup Isolation

Store backups in a separate AWS account that production credentials cannot access:

Production Account (111111111111)
        │
        │ AWS Backup copy rule
        ▼
Backup Vault Account (222222222222)    ← No production IAM can reach here
        │
        │ Vault Lock (immutable)
        ▼
    Logically Air-Gapped Vault         ← Cannot be deleted even by vault account admin

Configure with AWS Backup:

{
  "Rules": [
    {
      "RuleName": "DailyBackupWithCrossAccountCopy",
      "ScheduleExpression": "cron(0 2 * * ? *)",
      "TargetBackupVaultName": "production-vault",
      "Lifecycle": {
        "DeleteAfterDays": 35
      },
      "CopyActions": [
        {
          "DestinationBackupVaultArn": "arn:aws:backup:us-east-2:222222222222:backup-vault:isolated-vault",
          "Lifecycle": {
            "DeleteAfterDays": 90
          }
        }
      ]
    }
  ]
}

Layer 3: Immutable Storage with Vault Lock

AWS Backup Vault Lock enforces a WORM (Write Once Read Many) model — even an administrator with root access cannot delete or modify backups during the retention period:

aws backup put-backup-vault-lock-configuration \
  --backup-vault-name isolated-vault \
  --min-retention-days 7 \
  --max-retention-days 365 \
  --changeable-for-days 3

Once the changeable-for-days grace period expires, the lock becomes immutable — it cannot be removed, shortened, or overridden by any principal, including the root account.

Layer 4: Logically Air-Gapped Vault (Highest Protection)

For critical workloads, use AWS Backup's logically air-gapped vault — backups are locked by default, encrypted with AWS-owned keys, and isolated from your organizational control plane:

  • Cannot be deleted by any identity in the source account
  • Supports cross-account sharing for restore without granting delete permissions
  • Single-action cross-Region database snapshot copy (Aurora, Neptune, DocumentDB)
  • Multi-party approval for any administrative changes
aws backup create-logically-air-gapped-backup-vault \
  --backup-vault-name ransomware-resilient-vault \
  --min-retention-days 30 \
  --max-retention-days 365

Complete Ransomware-Resilient Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                        PRODUCTION ACCOUNT                                    │
│                                                                             │
│  ┌──────────────┐    replication    ┌──────────────┐                       │
│  │ Aurora Primary│ ───────────────→  │ Aurora Reader │  ← BOTH AT RISK      │
│  │ (us-west-2)  │                   │ (us-east-2)  │                       │
│  └──────┬───────┘                   └──────────────┘                       │
│         │                                                                   │
│         │ AWS Backup (PITR + scheduled snapshots)                           │
│         ▼                                                                   │
│  ┌──────────────┐                                                          │
│  │ Local Vault  │  ← Still at risk if account compromised                  │
│  │ (Vault Lock) │                                                          │
│  └──────┬───────┘                                                          │
└─────────┼───────────────────────────────────────────────────────────────────┘
          │ Cross-account copy
          ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                      ISOLATED BACKUP ACCOUNT                                 │
│                                                                             │
│  ┌────────────────────────────┐                                            │
│  │ Logically Air-Gapped Vault │  ← IMMUTABLE, isolated from prod IAM       │
│  │ • Vault Lock enforced      │                                            │
│  │ • Multi-party approval     │                                            │
│  │ • 90-day minimum retention │                                            │
│  │ • Cross-Region copy        │                                            │
│  └────────────────────────────┘                                            │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Validating Recovery with AWS Fault Injection Service (FIS)

Having backups is meaningless if you've never tested restoring them under realistic failure conditions. AWS FIS lets you simulate regional failures and validate your entire recovery chain — including your streaming analytics pipelines.

Test 1: Simulate Regional Database Failure

Use FIS to disrupt network connectivity to your primary Aurora cluster and verify:

  • PITR restore works from your cross-account vault
  • Your application correctly fails over to the secondary region
  • Data integrity checks pass after restore

Test 2: Validate Streaming Pipeline Resilience with Kinesis FIS Actions

If your architecture includes streaming analytics (Aurora CDC → Kinesis Data Streams → Glue/EMR/OpenSearch/Redshift), corruption doesn't stop at the database — it flows downstream. Use FIS native Kinesis Data Streams actions to test your pipeline's resilience:

# Create FIS experiment to inject ThrottlingException on Kinesis consumers
aws fis create-experiment-template \
  --description "Test analytics pipeline backpressure handling" \
  --targets '{
    "kinesisStream": {
      "resourceType": "aws:kinesis:stream",
      "resourceArns": ["arn:aws:kinesis:us-west-2:111111111111:stream/cdc-events-stream"],
      "selectionMode": "ALL"
    }
  }' \
  --actions '{
    "injectThrottling": {
      "actionId": "aws:kinesis:inject-api-error",
      "parameters": {
        "errorCode": "ProvisionedThroughputExceededException",
        "percentage": "80",
        "duration": "PT10M"
      },
      "targets": {"Streams": "kinesisStream"}
    }
  }' \
  --stop-conditions '[{"source": "none"}]' \
  --role-arn arn:aws:iam::111111111111:role/FISRole

What to validate during the experiment:

  • Does your Kinesis consumer (Lambda, KDA, Glue Streaming) handle ProvisionedThroughputExceededException with proper exponential backoff?
  • Do records land in a dead-letter queue or get silently dropped?
  • After the fault clears, does the pipeline catch up without duplicates or gaps in your downstream Redshift/OpenSearch analytics?
  • Monitor GetRecords.IteratorAgeMilliseconds — if it climbs indefinitely during the fault and never recovers, your pipeline has a data loss risk

Test 3: End-to-End Recovery Drill

Combine both tests into a quarterly recovery drill:

  1. Inject FIS network disruption to primary region (simulates regional impairment)
  2. Verify database failover — Aurora promotes secondary, application reconnects
  3. Verify streaming pipeline recovery — Kinesis consumers resume from checkpoint, no data loss in downstream analytics
  4. Run Athena validation queries against your Redshift/S3 lakehouse to confirm data completeness post-failover
# After failover, validate data completeness in Redshift via Athena
# Compare record counts between primary pipeline output and DR pipeline output
SELECT 
  DATE_TRUNC('hour', event_timestamp) AS hour,
  COUNT(*) AS record_count
FROM analytics_db.events
WHERE event_timestamp > CURRENT_DATE - INTERVAL '1' DAY
GROUP BY 1
ORDER BY 1;

Summary: The Minimum Viable Ransomware-Resilient Backup Strategy

RequirementImplementationWhy
Independent from replicationAWS Backup scheduled snapshots + PITRReplication propagates corruption
ImmutableVault Lock with compliance modePrevents attacker/admin from deleting backups
IsolatedCross-account copy to separate accountCompromised prod credentials can't reach backups
Air-gappedLogically air-gapped vaultHighest bar — even account admin can't delete
TestedFIS experiments + quarterly restore drillsUntested backups are Schrödinger's backups
Streaming validatedFIS Kinesis actions + pipeline catch-up verificationCorruption propagates beyond the DB into analytics
MonitoredAWS Backup Audit ManagerProve compliance, detect gaps before an incident

Key Takeaway

Multi-region replication protects against infrastructure failures — the physical plane. Immutable, isolated backups protect against logical corruption — the data plane. You need both. A ransomware attacker who gains write access to your primary database has, by design, write access to every replica — and every downstream streaming consumer (Kinesis, MSK, OpenSearch) faithfully processes the corrupted data. Only a backup that is architecturally separated from the write path survives, and only FIS testing proves your recovery actually works.

References

1 Comment

Well written and accurate... This is an important topic in this day and age. One small piece of information I would add is that backup retention periods can be different in different regions which may have a small benefit in cost reduction.

AWS

replied 24 days ago