Skip to content

RDS Instance restore stuck in "Creating" status

0

Hello,

I am experiencing a persistent issue where an RDS MariaDB 10.11 instance remains in the "Creating" status for an extended period when restoring from a recent snapshot.

Environment Details:

  • Engine: MariaDB 10.11.x
  • Source/Target Instance Class: m6g.4xlarge (also tested with m5.4xlarge and m6g.4xlarge with the same result).
  • Storage: gp3 with 12,000 IOPS and 1500 MiB/s throughput.
  • Snapshot Context: The snapshot contains a mix of InnoDB and MyISAM tables.
  • Account: Restoration is within the same AWS account and region.

Observation & Metrics: The Problem: The instance stays in "Creating" for several hours. Interestingly, an older snapshot of the same database restores successfully and quickly.

  • CloudTrail: I have reviewed CloudTrail events; the RestoreDBInstanceFromDBSnapshot API call is successful with no error messages.
  • CloudWatch Metrics (during "Creating" phase):
  • CPU Utilization: Consistently low (~3%).
  • DiskQueueDepth: Stalled at approximately 7.
  • ReadIOPS/Throughput: Very low, despite the high provisioned limits.

Suspected Root Cause: Given the presence of MyISAM tables, I suspect the engine is stuck in a mandatory myisamchk / index repair phase due to the snapshot being taken during an active state. Because of EBS "Lazy Loading" from S3, the single-threaded repair process is likely latency-bound, causing the low CPU/Queue Depth despite the high provisioned throughput.

Is there a way to verify from the RDS backend if a MyISAM repair is currently in progress? Are there any specific service-side throttles that affect MyISAM recovery during the "Creating" phase before the instance reaches "Available"?

Thank you.

3 Answers
1

While I cannot provide visibility into RDS backend processes or confirm specific service-side throttles during the "Creating" phase, I can address some aspects of your restore issue based on best practices for RDS snapshot restores.

Your observation about the snapshot containing MyISAM tables is significant. MyISAM tables can indeed cause restore failures or complications. When restoring from snapshots, temporary or MyISAM tables may not restore correctly because they aren't crash-safe and can become corrupted during the snapshot process. This is a known limitation that can prevent successful restoration.

To address slow restore performance in general, consider these approaches:

Optimize the source database before taking snapshots:

  • Avoid long-running transactions before snapshot creation, as they generate more undo records that must be processed during restore
  • Commit transactions in smaller batches to reduce undo log size
  • Create snapshots during low workload periods when there's less database activity

Improve restore performance:

  • Use a larger instance class with higher storage throughput and network bandwidth during the restore process
  • Monitor the RollbackSegmentHistoryListLength metric in CloudWatch on your source instance and keep it low, as high values indicate numerous undo records that can slow restores

Address the MyISAM table issue: The most reliable solution would be to convert MyISAM tables to InnoDB before taking snapshots, as InnoDB is crash-safe and handles snapshots more reliably. If MyISAM tables are causing the restore to fail or hang indefinitely, this conversion may be necessary for successful restores.

If you continue experiencing issues, you may need to work with AWS Support to investigate the specific restore behavior, as they would have access to backend logs and processes that aren't visible through standard monitoring tools.
Sources
Resolve Amazon Aurora MySQL snapshot restore issues | AWS re:Post

answered 9 months ago

EXPERT

reviewed 9 months ago

0

What you’re observing is consistent with engine-internal recovery, not an AWS infrastructure issue. Key points to clarify: • During the “Creating” phase, RDS does not expose engine logs, process state, or SHOW PROCESSLIST, so there is no supported way to verify whether myisamchk or MyISAM index repair is running. • If the snapshot contains MyISAM tables, MariaDB may perform mandatory table/index validation or repair before marking the instance Available. This happens inside the managed service layer and is opaque to customers. • There are no documented service-side throttles specific to MyISAM recovery. However, MyISAM repair is single-threaded and I/O-latency sensitive, which explains: • Low CPU • Flat DiskQueueDepth • Minimal read throughput despite high gp3 limits • The fact that older snapshots restore quickly strongly indicates snapshot state at capture time (e.g., MyISAM tables mid-write) rather than instance class or storage configuration. Practical guidance: • AWS Support is the only channel that can confirm backend recovery activity. • For future reliability, avoid MyISAM on RDS where possible, or ensure snapshots are taken after a clean shutdown / low-write window. • InnoDB-only snapshots do not exhibit this behavior. Bottom line: This is expected behavior when restoring MyISAM-containing snapshots, and there is no customer-visible signal or override during the Creating phase.

answered 9 months ago

0

Hey, ran into something very similar before so hopefully this helps. What's likely happening is MariaDB is running an internal myisamchk repair before the instance can come online. MyISAM doesn't have crash-safe recovery like InnoDB, so if the snapshot was taken while the instance was active it has to rebuild indexes from scratch. This runs single-threaded so your extra provisioned IOPS basically does nothing here. EBS lazy loading makes it worse blocks are being pulled from S3 on demand during the repair, which is why you see that stuck disk queue, near-zero CPU, and barely any actual IOPS. Classic signature of this problem. You can't verify it directly during the Creating phase but an AWS support ticket will get you host-level log confirmation pretty quickly. For next time, run FLUSH TABLES WITH READ LOCK before taking the snapshot gives MyISAM a clean state and cuts restore time dramatically. Also worth converting tables to InnoDB where possible, restore behavior is just way more reliable. If the restore keeps failing and you suspect actual file-level corruption, try some third party tools like Stellar Repair for MySQL it works directly on the raw .MYI and .MYD files without needing a running instance, which is really useful when the restore just won't complete at all. Hope this helps!

answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.