- Newest
- Most votes
- Most comments
The GenAI answer is partly correct. The first paragraph on crash recovery being delegated to storage and only needing to determine the LSN of the latest committed transaction rather having to perform roll forward / roll back on physical database blocks (pages) is correct. However, I wanted to add a few clarifying points around your statement that the documentation suggests "the new primary instance is created, which typically takes less than 10 minutes". First, could you please add a comment to your post and include the link to the documentation you reference so we improve the docs to better clarify this point.
Second, I would be neglegent not to reinforce that for business systems needing the highest levels of availability, we always recommend having at least one reader instance in a different AZ for failover to occur as quickly as possible. The minor versions we've recently released in the Dec'24 and March'25 timeframes have added a lot of optimizations to reduce failover and database startup time. Typically we see P50 statistics of failover times in the high single-digit seconds, not including DNS propagation. Topology-aware solutions like the AWS Advanced Driver wrappers (available for JDBC, ODBC, Node.js, Python, and Go) and RDS Proxy can further reduce time in half by eliminating DNS propagation entirely. Finally, it is most important to understand that different failure scenarios result in Aurora's control plane taking different actions.
Only in scenarios where physical hardware actually fails would the automation on the Aurora control plane would we need to provision a new instance. It is the provisioning of new hardware that typically takes on the order of minutes. If there is a healthy reader instance, the writer role will failover to it quickly before replacing the failed hardware and bringing it back into the cluster topology as a reader node. Also, our health monitoring layers can detect software component failures and in many of those scenarios can simply restart the component, reestablishing availability in a few seconds. This would be true in the case of a database crash from things like a software bug or an out-of-memory (OOM) condition. Even things like a manual database restart to change a static parameter like "max_connections" now only takes a few seconds.
I hope this clarifies our operational behavior and dispels the myth that any failure requires provisioning new hardware.
Hello PThawley,
Thank You very much for your answer! Let me clarify the case. We have hundreds of instances in production all of which have dual - writer/reader setup. Overtime we realized that some of the services can tolerate downtime from minute to half an hour. Aurora having the ability to recover from hardware failure presents good opportunity to save cost by weighting the risk per service to run a single node db. We were hoping someone could share experience with single node hardware failure so that we don't count all cases against 10 minutes downtime. And here is the link to the doc Thanks again! Rado
answered a year ago
Amazon Aurora is designed to recover from unplanned restarts almost instantaneously. Unlike traditional databases, Aurora's architecture decouples storage and compute, which provides significant benefits during recovery. When an Aurora instance restarts after a crash, it doesn't have to go through the traditional data recovery process of replaying transactions in the redo log. Instead, Aurora delegates recovery to the Aurora Storage layer, which only needs to determine the point where the storage volume is transactionally consistent.
However, for a single Aurora instance (without replicas), there is still time needed to provision a new instance on failure, which results in a longer Recovery Time Objective (RTO). To reduce this recovery time to seconds, you can deploy Aurora replicas in additional AWS Availability Zones.
The recovery time for an Aurora instance depends on several factors:
- The crash recovery process itself
- Database activity at the time of the restart
- The specific behavior of your DB engine
To improve reboot time, it's recommended to reduce database activity as much as possible during the reboot process, as this reduces rollback activity for in-transit transactions.
If you have binary logging enabled, this will directly affect recovery time after an unplanned restart because it forces the DB instance to perform binary log recovery. The amount of binary log data affects recovery time - more logged data means the instance must process more during recovery, increasing recovery time.
For critical workloads requiring minimal downtime, consider implementing Aurora replicas across multiple Availability Zones. With replicas, failover typically completes within 30 seconds, and using Amazon RDS Proxy can further reduce failover time by 66% while preserving application connections.
Sources
Amazon Aurora for Core Banking Systems | AWS for Industries
Amazon Aurora reliability - Amazon Aurora
Amazon Aurora updates - Amazon Aurora
Database migration considerations - Migrating Your Databases to Amazon Aurora
answered a year ago
Relevant content
asked 3 years ago
Thanks for the clarification Rado. Your reasoning makes perfect sense to me as a cost optiization. It is normal in most large enterprises for different systems to have different RPO and RTO requrements so I applaud your work align your deployments accordingly. In our documentation, we are often quite conservative when quoting numbers like in this case where we say "Service is restored when the new primary instance is created, which typically takes less than 10 minutes".
The control plane workflow to perform a "host replacement" is effectively the same workflow as adding a reader instance to an existing cluster. My own experience in working with customers is that this typically completes in about 3-5 minutes. Unfortunately, the natural entropy in a public cloud make long-lived workflows like this subject to some degree of variance, which is why the docs say "typically less than 10 minutes".
You should also keep in mind there are at least a couple failure scenarios where host replacement might fail. For example, consider a complete network partition event where the AZ is fully inaccessible. Host replacement only replaces the instance in the same AZ as the original one so if your current AZ is completely dead, replacement will fail and you'll need to manually add an instance in another AZ. The same would be true if there is no more free capacity of the instance type/size of your original instance. 😰
Hope this helps!