By using AWS re:Post, you agree to the AWS re:Post Terms of Use

Copy from S3 to RDS Restore Instance Limited @ 100MB/s Regardless of Instance & Storage Limits

0

I am testing out a backup and restore process. I have a single 300+GB file in an S3 bucket and am using that file as the source for the RDS restore from backup wizard/process.

Whether I use a t3 or r5 instance, we hit a consistent 100MB/s / 1Gbps limit on the copy from S3 to RDS instance phase.

Is there a specific limit to the RDS restore from backup process, because our instance and storage limits are well within bounds but the transfer is limited to 100MB/s / 1Gbps regardless of what type of instance we utilize.

The S3 bucket and the RDS restore instance targets are all in the same region.

I can send the backup file from the third-party to AWS over public internet much faster than the restore process can copy the file from S3 to RDS. Any thoughts on what the issue may be?

Thank you.

1 Answer
0

The issue you're experiencing with the 100MB/s limit during the RDS restore process from an S3 backup is not explicitly mentioned in the provided sources. However, based on the information available, there are a few points to consider:

  1. RDS integration with S3 has some inherent limitations. While these don't directly mention a 100MB/s transfer limit, they do indicate that there are constraints on how RDS interacts with S3.

  2. The restore process you're using might be different from the native backup and restore functionality that's available for certain RDS instances, like RDS Custom for SQL Server 2022. This native functionality allows for more direct interaction with S3 buckets.

  3. The consistent 100MB/s limit across different instance types (t3 or r5) suggests that this might be a service-level limitation rather than an instance-specific one.

  4. The fact that you can transfer the file to AWS over the public internet faster than the internal S3 to RDS copy is unusual, as typically internal AWS network transfers are faster than external ones.

Given these points, here are some suggestions:

  1. Check if your RDS instance type supports native backup and restore with S3 integration. If it does, this might provide better performance.

  2. Consider splitting your large 300+GB file into smaller chunks. Some RDS features have file size limitations (e.g., 50GB for uploads from RDS to S3).

  3. Reach out to AWS Support for clarification on any specific limits for the RDS restore from backup process, as this might be a service-specific limitation not publicly documented.

  4. Explore alternative backup and restore methods that might offer better performance, such as using AWS Database Migration Service (DMS) or other AWS-native tools designed for large data transfers.

Remember that while these suggestions might help, the specific 100MB/s limit you're encountering isn't directly addressed in the available information, and might require further investigation with AWS support.
Sources
Native backup and restore with Amazon S3 integration on Amazon RDS Custom for SQL Server 2022 | AWS Database Blog
Integrating an Amazon RDS for SQL Server DB instance with Amazon S3 - Amazon Relational Database Service

profile picture
answered 10 days ago
profile picture
EXPERT
reviewed 8 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions