Skip to content

AWS DataSync vs Batch operations for S3 cross regions

0

For large volume of data to be copied from bucket 1 in region 1 to bucket 2 in region 2 which one is fastest?

asked 2 years ago1.1K views

1 Answer
1
Accepted Answer

I just share highlights:

Depends on your object size and number both can be a great solutions.

  • I don’t recommend Transfer Accelerator for bucket to bucket transfer as it adds extra network hops and possible delays.
  • AWS Batch can be a great way to copy/replicate from 1 region to another.
  • If you have objects greater than 5Gb batch replication will be easier than copy.
  • AWS DataSync can have some quota limits.
  • Batch Operations Jobs take up to four phases to fully complete: (optional) manifest generation, preparation, execution, and (optional) report generation. so the performance estimates for each part of the job comes with a TPS estimate . You can estimate the total seconds each phase will take by dividing the total number of objects in their job by the TPS estimate.

So here are the comparison:

AWS DataSync limits: https://docs.aws.amazon.com/datasync/latest/userguide/datasync-limits.html

AWS Batch Operations: https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-examples-copy.html

AWS
EXPERT

answered 2 years ago

EXPERT

reviewed 2 years ago

EXPERT

reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.