The idiomatic approach is S3 object replication, and in particular cross-region replication. You can copy objects to another bucket in the same region or another region, and in the same account or another account. You can also copy to multiple target buckets. You can contact AWS support to copy existing objects. Do be aware of S3 pricing for replication as you will still incur data OUT transfer charges which can be quite costly.
For a one-and-done experience, see the
aws s3 sync CLI command or use
s3cmd sync. Data transfer pricing rules still apply. Note that this operation can be very slow for buckets with many objects (millions). Also, these tools first copy from S3 to your system, then from your system to S3, unlike replication which is a direct bucket-to-bucket transfer (or at least, that's how it appears to the user).
AWS provides multiple tools to copy objects between your buckets.
Use AWS DataSync for ongoing data distribution, data pipelines, and data lake ingest, as well as for consolidating or splitting data between multiple buckets.
Use S3 Replication for continuous replication of data to a specific destination bucket.
Use S3 Batch Operations for large-scale batch operations on S3 objects, such as to copy objects, set object tags or access control lists (ACLs), initiate object restores from Amazon S3 Glacier, invoke an AWS Lambda function to perform custom actions using your objects, manage S3 Object Lock legal hold, or manage S3 Object Lock retention dates.
As for retaining permissions:
With S3 Replication
By default, the owner of the source object also owns the replica. Amazon S3 replicates the object version and the ACL associated with it.
If you add the owner override, Amazon S3 replicates only the object version, not the ACL. In addition, Amazon S3 doesn't replicate subsequent changes to the source object ACL. Amazon S3 sets the ACL on the replica that grants full control to the destination bucket owner. When you update a replication configuration to enable, or disable, the owner override, the following occurs.
If you add the owner override option to the replication configuration:
When Amazon S3 replicates an object version, it discards the ACL that is associated with the source object. Instead, it sets the ACL on the replica, giving full control to the owner of the destination bucket. It doesn't replicate subsequent changes to the source object ACL. However, this ACL change doesn't apply to object versions that were replicated before you set the owner override option. ACL updates on source objects that were replicated before the owner override was set continue to be replicated (because the object and its replicas continue to have the same owner).
If you remove the owner override option from the replication configuration:
Amazon S3 replicates new objects that appear in the source bucket and the associated ACLs to the destination buckets. For objects that were replicated before you removed the owner override, Amazon S3 doesn't replicate the ACLs because the object ownership change that Amazon S3 made remains in effect. That is, ACLs put on the object version that were replicated when the owner override was set continue to be not replicated.
With S3 Batch
S3 Batch Operations support custom ACLs that you define and canned ACLs that Amazon S3 provides with a predefined set of access permissions.
If the objects in your manifest are in a versioned bucket, you can apply the ACLs to specific versions of every object. You do this by specifying a version ID for every object in the manifest. If you don't include a version ID for any object, then S3 Batch Operations applies the ACL to the latest version of the object.
If the objects in the manifest are in a bucket uses the bucket owner enforced setting for Object Ownership, the Replace access control list (ACL) operation can only specify object ACLs that grant full control to the bucket owner. The operation can't grant object ACL permissions to other AWS accounts or groups. For more information, see Controlling ownership of objects and disabling ACLs for your bucket.
When transferring objects between Amazon S3 buckets, DataSync copies objects together with object metadata and tags. DataSync does not copy other object information such as object ACLs or prior object versions.
Move files from S3 Bucket back to Siteasked 7 months ago
Where can I found the uri of my Bucket to connect my Symfony application with S3 and upload files?asked 6 months ago
How can we set s3 lifecycle rule to delete the files form S3 bucket!asked 5 months ago
Redshift Copy fails due to '\' present in some of the data in my csv fileasked 2 years ago
How do I copy files from my S3 bucket hosted in Europe to my S3 bucket hosted in the United States?asked a year ago
Failed to COPY parquet files from S3 to Redshiftasked 2 months ago
Cross account copy from S3 to Deep ArchiveAccepted Answerasked 2 years ago
Copying earlier version of object in S3 bucket to Redshift with COPY commandasked 2 months ago
Batch download files from multiple different folders in the same S3 bucketasked 7 months ago
Unable to copy elasticache for redis backup to S3 bucket in the same regionAccepted Answerasked a month ago