transfers between S3 buckets that are each in different companies

Question

Trying to get a definitive answer on something that I might just be missing. I work for a bank that has been asked by a vendor to do a S3-to-S3 bucket transfer of data (pretty small). What I'm trying to determine is:

- Will this type of transfer require a VPC gateway or Private Link if we don't want the endpoints to be public facing?
- If we do this type of transfer, I assume it will stay within the AWS infrastructure and not be traversing the public internet (assuming set up properly)
- Are there other things to consider? cost (should be small files)? stability?

Thank you for entertaining a noob question.

Serge · Accepted Answer

Hello

When copying data between Amazon S3 buckets within the same region, the transfer remains inside the AWS network. AWS relies on its high-speed internal infrastructure to move data between services, ensuring the process is fast, secure, and cost-efficient. Any copy operation you initiate is handled automatically across this private network.

If you copy data across regions (or to destinations outside AWS), the transfer will instead travel over the public internet, which can result in additional data transfer charges.

To maintain private connectivity between regions, AWS recommends using VPC peering or AWS Transit Gateway. With Transit Gateway, you can also set up VPC endpoint attachments to enable private endpoint access.

Best Regards

Leo K · Answer

You don't need Transit Gateways or other complexities. Copying objects between buckets internally within the S3 platform and AWS infrastructure is done by calling the CopyObject API of S3. You can do that with the AWS CLI, for example, with the command "aws s3 cp" documented here: https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html. There's a more comprehensive discussion of copying objects in this documentation section: https://docs.aws.amazon.com/AmazonS3/latest/userguide/copy-object.html

The CLI command could look like this. The `--dryrun` parameter causes the operation only to be simulated. Remove the parameter when you're comfortable with the simulation results:
```bash
aws s3 cp s3://amzn-source-bucket/sourcefolder/ s3://amzn-target-bucket/ --recursive --dryrun
```

Importantly, if the source and destination buckets are in different AWS accounts, the credentials (such as an IAM role) that you use to call the CopyObject API must have read permissions (usually `s3:ListBucket` and `s3:GetObject`) to the source bucket and objects within, as well as write permissions (`s3:ListBucket` and `s3:PutObject`) to the destination bucket. If either bucket or both are using KMS encryption (usually shown as the "SSE-KMS" default encryption type in the bucket's properties), the IAM role or user you use to call CopyObject must also have the permissions `kms:Decrypt` and `kms:GenerateDataKey` to the KMS key(s) of the bucket(s).

If you want the call to the CopyObject API call also to stay completely within the AWS infrastructure, one way is to make the AWS CLI call via AWS CloudShell in the management console. No VPCs, transit gateways, or anything else is needed for that. Even if the buckets are in different regions, traffic remains within the AWS infrastructure, using AWS's backbone network for both intra-region and inter-region connectivity.

To be clear, while easier to set up, what you won't want to do is first to download a copy of the objects to your local system (outside AWS) and then running a second operation to upload them to the second bucket. This is often more straightforward, because the download operation can use one set of credentials and the upload operation a second set. However, the objects would be downloaded to your local system, violating your stated requirement to keep the data transfer internal to the AWS infrastructure.

transfers between S3 buckets that are each in different companies

Add your answer

Relevant content