Questions tagged with Disaster Recovery/Business Continuity

I had setup AWS Backup some time ago for RDS and, as far as I could tell, it was correct. I was able to restore from it in my tests (from both vaults - one for production with WORM setup, and one non-production vault for resources that are used for development). The backups were also running correctly - I would see them being made each day for the Snapshots and Continuous also looked correct. Just recently - mid last week - I have not been able to see or access backups. When I view the Backup Vaults as the root user of the account, I no longer see any backups from a certain date forward (Creation Time doesn't go past May 12th). When I look at the specific protected resources, I can see creation times sooner than that, but the backups are all prefixed with `rds:` rather than `awsbackup:`. Given that AWS Backup is managing the backups (or should be) this doesn't make sense to me. Additionally, when I actually go to attempt a restore from the console with ANY of these, I receive an error that says: "Failed to retrieve snapshot." Again, I am using this as a root user and it did work previously. Very concerned about this. I would appreciate any help or guidance.

We are considering a move of our corporate data to AWS (that will then reside in a PostgreSQL DB), however we would like to maintain a backup of the data to one of our own servers residing outside the AWS cloud (in our own data center). Is this possible?

I'm exploring Aurora MySQL write forwarding and need a couple of clarifications. The goal is to run the application active-active in 2 regions. The app in region 1 (primary) will connect to the writer endpoint in region 1 while the app in region 2 (secondary) will connect to the reader endpoint of the secondary cluster in the Aurora global database. The idea is to use write forwarding in region 2 to send the writes to the primary cluster. Now for the questions 1. What is the best way to set "aurora_replica_read_consistency" for all the sessions connecting to the secondary Aurora cluster? In case of a failover I'm guessing we will need to unset this parameter in region 2 and set it in region1. 2. Also in the case of a failover, the endpoints will need to changed in both the regions. The application in region 1 will need to be updated to use the reader endpoint, while the application in region 2 will need to be updated to use the writer endpoint. What is the best way to automate this steps?

Two regions, S3 replication configured with replica modification sync on the prefix where Hudi dataset is located. Hudi writes are exclusive to a single region. - will S3 replication maintain consistent Hudi dataset in the replicated region ? - If we move Hudi writes to the replication region (failover), will hudi dataset stay consistent in the original region, maintained by replica modification sync (failback) from region 2?

Hello guys! I was able to successfully recover my instances after the outage in the premise office, currently, all my machines are running in EC2 and source servers are shown as "Ready, lag 9d" "Stalled" and I keep being charged for replica EBS volumes, as well as AWS Replication Instances (t3.small). While I don`t have any info on when it will be possible to perform the failback, I`d like to cut some costs and remove replica EBS volumes. I was thinking about the action "Disconnect from AWS" but it does not work with the message: > Some Source servers could not be processed: > s-1ed6*****9: Cannot disconnect Source Server s-1ed******9 because it has a Recovery Instance. What would be the correct action in such a case? (remove source server\remove just the volumes\something else) (the goal to cut the costs while keeping the possibility of failback later). Thanks! Regards,

Hello, We are currently using Aurora MySQL Regional Cluster and a writer instance in US-EAST-1 region without Multi AZ enabled. I am trying to enable Multi AZ deployment for my existing RDS cluster and don't see an option to configure under Cluster or Instance via AWS Console or CLI. Thanks Sandeep G

Hi Everyone, Last year, We moved one of our main reporting systems to Elasticseach 7.9 , its located in N. California region. In this region only 2 AZ is available, in our domain we have dedicated 3 master nodes. Now we are looking for some disaster recovery options. I read https://docs.aws.amazon.com/opensearch-service/latest/developerguide/disaster-recovery-resiliency.html, and it looks with our current setup we have 50/50 downtime possibility. so far I came up with two options 1- cross cluster replication(which will double our cost) 2- continuous snapshot backup. Below are my questions 1- Is there any other possible option for disaster recovery that I haven't considered? 2- For snapshot backup, is there an easy way to schedule/manage the backups, so we can bring up the domin in shortest period of time and with less data loss.

Hi Everyone, Last year, We moved one of our main reporting systems to Elasticseach 7.9 , its located in N. California region. In this region only 2 AZ is available, in our domain we have dedicated 3 master nodes. Now we are looking for some disaster recovery options. I read https://docs.aws.amazon.com/opensearch-service/latest/developerguide/disaster-recovery-resiliency.html, and it looks with our current setup we have 50/50 downtime possibility. so far I came up with two options 1- cross cluster replication(which will double our cost) 2- continuous snapshot backup. Below are my questions 1- Is there any other possible option for disaster recovery that I haven't considered? 2- For snapshot backup, is there an easy way to schedule/manage the backups, so we can bring up the domin in shortest period of time and with less data loss. Thanks in advance.

Hello guys! I`ve configured the AWS DRS service in eu-central-1 region. There are few servers in sync with the status "Ready", but when I`m trying to initiate the recovery or drill - the console throws an error "Failed to initiate drill. One or more of the source servers initiated are currently recovering or launching" While checking the current jobs (both in UI and CLI) there are no active recovering or launching instances. Any ideas on how to solve it? Thanks! Regards,

I did a lab on AWS elastic disaster recovery in my aws free tier account. After I was done with lab I cleared all the resources. But still a snapshot in name of EDR Recovery is being created everytime. Even when I delete it, it comes up again. I think it is because of replication setting. If anyone can suggest how to delete this replication setting it would be a great help

If my cluster is setup as follows: Brokers: 3 AZ: 3 RF: 3 MinISR: 1 Ack: all Q1: If a broker is being upgraded, Kafka will reassign the leadership of some partitions. After the upgrade will the leaderships get reassigned again so that all brokers are being used as before? Q2: If 1 AZ (AZ1) goes down, I understand that Kafka will automatically reassign the partitions to the other brokers in the two AZs without impacting the producers and consumers. When AZ1 comes back will MSK **automatically** create/restart the failed broker and redistribute the partitions?

I have a Aurora db cluster with 2 instance, a writer and a reader. Since I migrated from version 1.23 (mysql 5.6) to 2.10.1 (mysql 5.7) I had some failover. Could you help me to understand what can cause this behavoral? What event can trigger a automatic failover?

There is a 4 years old thread on aws forums but i think there is still no support for multi-region cognito? https://forums.aws.amazon.com/thread.jspa?messageID=777006 we run on us & asia regions and cognito is the single bottleneck. We can run databases, APIs etc cross-region but not cognito :( One thing is lag. We can either support US or Asia customers better now :( But another high topic is disaster recovery & failover. This month was real bad for AWS, it was down in US regions for 3 times in a month. Last year, AWS was also down during Q4 of 2020 and Q4 is most important Q of the year for many because of holiday sales. We run on n.virginia (probably most pop region?) and it was heavily affected during this December, when the traffic is at it's peak and everyone just looks how AWS region is down and $ is not collected from the peak sales period. I bet a lot of you had that frustration during this December 2021 too. And also 2020, while AWS was down during Q4 of 2020 too. AWS talks constantly - everything fails, do multi region. Follow the 5 pillars. How can we follow AWS best practises when core service which is in use by user for any API call can not run by AWS's best practises? What's the point to run database in multi-region as failover if user can't access it anyways because of the 1st service it needs to hit is single-region and it is down? The thread around this started back in 2017 Mar, It's been 4 years already and still no option to run multi-region authorization. AWS, Is there any progress on this?

Based on the recent outage at us-east-1, where we have most of our AWS resources, the company has taken the path to start transitioning into a multi-region architecture, but picking which extra region(s) to use is not a matter of chance and the decision must be thoughtfully taken. Right now the first criteria to consider is cost as most US-based regions are cheaper for example than the one in Sao Paulo. Latency for our customers (distributed across LATAM and US) might be worthwhile considering, but right now everything is working from Virginia. For major disasters maybe replicating to a different country, or perhaps even continent could be the best for geographic isolation, but at a cost. How do you actually decide which regions to use? How to weigh different factors? Is there an actual guide for such?

Can an application be architected so that it leverages CloudHSM clusters in multiple regions (at least 2)? Possibly by using the cross-region replication/cloning so data can be decrypted in both regions if needed. Assumption is that CloudHSM enveloping keys do not have a region affinity much like KMS CMKs.

Do we have a RDBMS solution in combination with other features/products/services that we can offer to our customer, guaranteeing Zero RPO across regions?

A customer wants to automate the backup for S3 buckets, EBS snapshots and RDS snapshots to another, independent account to be able to restore the application data in case an administrator account in the organization is compromised and a ransomware attack is executed. The customer wants to do this in an automated, maintenance free way. At first I suggested using scheduled Lambdas in the independent accounts that use IAM roles to access the "to-be-backuped" buckets and snapshots and pull them into the independent account. However, this solution requires the implementation and maintenance of code. I was looking into AWS backup as I thought it would be able to create RDS and EBS Snapshot backups on a schedule to S3. I would then have recommended the customer to use this and setup cross-account replication of the buckets with a transfer of ownership of the objects in the replicated bucket to the independent account. However, it seems that AWS backup uses S3 as a storage location for the backups, the backups themselves are not visible/accessible this way. I am looking for a low effort, maintenance free way of achieving cross account (destination account being outside of the org) backups for S3, EBS, and RDS

Customer want to start redirecting a small % of the traffic to AWS and the rest keep it on-prem, so they can learn and improve the platform. They will gradually increase the traffic until it reach 100%. How can they be assured using just Route 53 with weighted routing policy that the users that get directed to AWS always (while they are on the same IP) goes to the AWS platform and not to on-prem on the next DNS resolution?. They need to keep the sessions of the user on the first destination it was resolved. Does Route 53 weighted routing makes any sort of stickiness like other 3rd-party Load Balancers are able to do?.

Customer is storing Oracle RMAN backup on-prem. As part of DR strategy, they want to build DR environment in AWS. I would like them to use RDS in AWS for their DR instance. Is there way to spin up RDS instance using RMAN backup and then later use RDS snapshot to restore back to on-prem Oracle ?