By using AWS re:Post, you agree to the Terms of Use

How do I copy files from my S3 bucket hosted in Europe to my S3 bucket hosted in the United States?


I'm trying to copy my files from our partners in Europe to our site here in the United States. I am supposed to have full access to these files when I am done. How do I do this?

2 Answers

The idiomatic approach is S3 object replication, and in particular cross-region replication. You can copy objects to another bucket in the same region or another region, and in the same account or another account. You can also copy to multiple target buckets. You can contact AWS support to copy existing objects. Do be aware of S3 pricing for replication as you will still incur data OUT transfer charges which can be quite costly.

For a one-and-done experience, see the aws s3 sync CLI command or use s3cmd sync. Data transfer pricing rules still apply. Note that this operation can be very slow for buckets with many objects (millions). Also, these tools first copy from S3 to your system, then from your system to S3, unlike replication which is a direct bucket-to-bucket transfer (or at least, that's how it appears to the user).

answered a year ago
reviewed 10 months ago
reviewed 10 months ago

AWS provides multiple tools to copy objects between your buckets.

Use AWS DataSync for ongoing data distribution, data pipelines, and data lake ingest, as well as for consolidating or splitting data between multiple buckets.

Use S3 Replication for continuous replication of data to a specific destination bucket.

Use S3 Batch Operations for large-scale batch operations on S3 objects, such as to copy objects, set object tags or access control lists (ACLs), initiate object restores from Amazon S3 Glacier, invoke an AWS Lambda function to perform custom actions using your objects, manage S3 Object Lock legal hold, or manage S3 Object Lock retention dates.

As for retaining permissions:

With S3 Replication

By default, the owner of the source object also owns the replica. Amazon S3 replicates the object version and the ACL associated with it.

If you add the owner override, Amazon S3 replicates only the object version, not the ACL. In addition, Amazon S3 doesn't replicate subsequent changes to the source object ACL. Amazon S3 sets the ACL on the replica that grants full control to the destination bucket owner. When you update a replication configuration to enable, or disable, the owner override, the following occurs.

If you add the owner override option to the replication configuration:

When Amazon S3 replicates an object version, it discards the ACL that is associated with the source object. Instead, it sets the ACL on the replica, giving full control to the owner of the destination bucket. It doesn't replicate subsequent changes to the source object ACL. However, this ACL change doesn't apply to object versions that were replicated before you set the owner override option. ACL updates on source objects that were replicated before the owner override was set continue to be replicated (because the object and its replicas continue to have the same owner).

If you remove the owner override option from the replication configuration:

Amazon S3 replicates new objects that appear in the source bucket and the associated ACLs to the destination buckets. For objects that were replicated before you removed the owner override, Amazon S3 doesn't replicate the ACLs because the object ownership change that Amazon S3 made remains in effect. That is, ACLs put on the object version that were replicated when the owner override was set continue to be not replicated.

With S3 Batch

S3 Batch Operations support custom ACLs that you define and canned ACLs that Amazon S3 provides with a predefined set of access permissions.

If the objects in your manifest are in a versioned bucket, you can apply the ACLs to specific versions of every object. You do this by specifying a version ID for every object in the manifest. If you don't include a version ID for any object, then S3 Batch Operations applies the ACL to the latest version of the object. .
If the objects in the manifest are in a bucket uses the bucket owner enforced setting for Object Ownership, the Replace access control list (ACL) operation can only specify object ACLs that grant full control to the bucket owner. The operation can't grant object ACL permissions to other AWS accounts or groups. For more information, see Controlling ownership of objects and disabling ACLs for your bucket.

With DataSync

When transferring objects between Amazon S3 buckets, DataSync copies objects together with object metadata and tags. DataSync does not copy other object information such as object ACLs or prior object versions.

answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions