Help with copying s3 bucket to another location missing objects
Hello All, Today I was trying to copy a directory from one location to another, and was using the following command to execute my copy.
aws s3 s3://bucketname/directory/ s3://bucketname/directory/subdirectory --recursive
The copy took overnight to complete because it was 16.4TB in size, but when I got into work the next day, it was done, or at least it had completed.
But when I do a compare between the two locations I get the following
bucketname/directory/ 103,690 objects - 16.4TB
bucketname/directory/subdirectory/ 103,650 - 16.4TB
So there is a 40 object difference between the source location and the destination location.
I tried using the following command to copy over the files that were missing
aws s3 sync s3://bucketname/directory/ s3://bucket/directory/subdirectory/
which returned no results. It sat for a while maybe like 2 minutes or so, and then just returned to the next line.
I am at my wits end trying to copy of the missing objects, and my boss thinks that I lost the data, so I need to figure out a way to get the difference between the source and destination copied over.
If anyone could help me with this, I would REALY appreciate it. I am a newbie with AWS, so I may not understand everything that I am told, but I will try anything to get this resolved. I am doing all the commands through an EC2 instance that I am ssh into, and then use AWS CLI commands.
Thanks to anyone who might be able to help me.
Take care,
-Tired & Frustrated :)
I can't answer for the discrepancy between the number of objects. Rather than using an EC2 instance to synchronize the buckets, please check out the resources below.
Relevant questions
Access denied when trying to GET objects uploaded to s3 bucket via aws sdk using cloudfront
asked 6 months agoCross Account Copy S3 Objects From Account B to AWS KMS-encrypted bucket in Account A
asked 4 months agoCopying labeled dataset for AWS Rekognition from one account to another
asked 5 months agoHelp with copying s3 bucket to another location missing objects
asked 10 days agoIs there a way to not change an S3 object's Last-Modified date when copying it to a new bucket?
Accepted Answerasked 2 years agoManually set S3 replication status
Accepted Answerasked 2 years agoNewb questions... Copy from 1 bucket to another
asked 2 years agoSync to me-south-1 from eu-west-1 error!
asked 3 years agoCopy data from EFS to EFS in another AWS account.
asked 2 years agoGetting access denied when following the documentation on how to copy objects between Lightsail buckets
Accepted Answerasked 4 months ago