Copy data cross account cross region using DataSync
Hi AWS, I am trying to copy data from S3 bucket in one region in source account to another S3 bucket in another region in destination account. I was able to create the DataSync locations successfully for both source and destination, but now when I am creating the task to test whether I am able to copy data from source bucket to destination bucket across accounts and regions I am not able to do and while I checked the DataSync Log Group, it is giving the error Execution exec-056dfghgdxxxxxx finished with status Unable to connect to S3 endpoint.
I know it is possible to transfer data across accounts for the same region using DataSync, but is it possible in case of different regions as well. I am really confused. Please help
Details in case you get stuck because of the description provided above:
- Source account: 111111111111
- Destination account: 222222222222
- Source bucket: datasync-source-bucket
- Destination bucket: datasync-dest-bucket
- Source region: ca-central-1
- Destination region: eu-west-2
- Newest
- Most votes
- Most comments
Here is an updated tutorial in the AWS DataSync documentation for transferring data cross account and cross region with AWS DataSync. https://docs.aws.amazon.com/datasync/latest/userguide/tutorial_s3-s3-cross-account-transfer.html
This tutorial notes that you must create and start your DataSync task from the region of the destination S3 location. In this case the task would be created and run in the destination region: eu-west-2
Step 6.2
Important
To avoid a network connection error, you must start your DataSync task from the Region of the destination location.
Hi,
You have a step-by-step implementation for cross-region and cross-account with Datasync in this blog post: https://aws.amazon.com/blogs/storage/transferring-file-data-across-aws-regions-and-accounts-using-aws-datasync/
Just modify their values to yours to get your system to. work
Best,
DIdier
Ensure the data sync agent has access to the internet via a NAT gateway so that it can access the S3 endpoints in the different regions.
The agent needs to be on a private subnet which has a route to a NAT gateway.
Relevant content
- asked 7 months ago
- Accepted Answerasked 10 months ago
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 months ago
I am still getting the error Unable to connect to S3 endpoint. So the point I forgot to mention is both my source and destination buckets are encrypted with Customer Managed Key (CMKs). I have even updated the key policy but it's working for the same region cross account but not for cross region cross account.
Hi Arjun, When using a Customer Managed Key (CMK) for cross account data transfers you must specify the SSE-KMS key in the IAM role DataSync is using to access the Amazon S3 bucket. In addition, the IAM role must be specified in the SSE-KMS key policy [1]. Cross-account key access documentation [2]. You should also validate you are starting the DataSync task in the destination region.
[1] https://docs.aws.amazon.com/datasync/latest/userguide/create-s3-location.html#create-s3-location-encryption [2] https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying-external-accounts.html