S3 batch replication Performance

Question

We are trying to do an S3 batch replication of files from a bucket in me-south-1 to a bucket in me-central-1.
After 3 hours in active state, the job is only 2.8% done with only 300k files out of 12 million files replicated.

This is much slower than we expected - and to me it looks like the transfer speed has gets slower the longer the job runs.
Looking at evidence online, the batch replication should be much faster than this. Is there something we can do to make this faster?

re:Post Agent · Answer

The performance you're experiencing with S3 batch replication seems slower than expected. There are several factors that could be affecting the speed of your replication, and there are steps you can take to potentially improve it:

1. Object size and number: The performance of batch replication can vary depending on the size and number of objects. If you have a large number of small files, it might take longer than fewer, larger files.

2. Network bandwidth: Ensure that there are no network bottlenecks between the regions. While me-south-1 and me-central-1 are relatively close geographically, network conditions can still impact transfer speeds.

3. S3 request rates: There might be limits on the number of requests per second that S3 can handle. You could check if you're hitting any service quotas and request an increase if necessary.

4. Parallel processing: S3 batch operations are designed to work in parallel, but the degree of parallelism might need adjustment. You could try creating multiple smaller batch jobs instead of one large job to increase overall throughput.

5. S3 Transfer Acceleration: While not specifically for batch operations, enabling S3 Transfer Acceleration might help improve cross-region transfer speeds.

6. Job configuration: Review your batch job configuration to ensure it's optimized for your use case. You might need to adjust parameters like priority or description to better suit your needs.

7. Monitoring and metrics: Use Amazon CloudWatch to monitor the progress of your batch job and identify any bottlenecks or issues that might be slowing it down.

8. Alternative methods: If batch replication continues to be slow, you might want to consider alternative methods like using AWS DataSync for large-scale data transfers between S3 buckets in different regions.

Remember that the initial stages of a large replication job might seem slower as the system sets up the necessary resources. However, if the speed continues to decrease over time, it would be worth investigating further or contacting AWS support for more specific guidance tailored to your account and use case.

If you've already considered these factors and are still experiencing unusually slow performance, it may be beneficial to reach out to AWS support for a more in-depth analysis of your specific situation.
      
**Sources**  
[AWS DataSync vs Batch operations for S3 cross regions | AWS re:Post](https://repost.aws/questions/QULj1ONZknT52Djm2aTj8Qjw/aws-datasync-vs-batch-operations-for-s3-cross-regions)  
[S3 Transfer Rates Capped at 2.8mb/s ??? How Can I Speed This Up? | AWS re:Post](https://repost.aws/questions/QUUdZKwva0TO6EwrPcFLtErA/s3-transfer-rates-capped-at-2-8mb-s-how-can-i-speed-this-up)

SHAJAM · Answer

Can you clarify the size of all the files combined. S3 does have a limit on how fast the data will be transferred. You can request to increase this limit.

`AWS Console > Service Quota > S3 > Replication transfer rate. `

Note, this might require approval from AWS Support team depending on your requests.

S3 batch replication Performance

Relevant content