Size of all files and folders moved to S3

0

Hello All,

I've a requirement to move 4 TB of data from on-prem shared file system to AWS S3. We can't affort to miss any files during this movement. Now data has been moved to S3 - but my client need a confirmation that all the files are moved from on-prem to S3 successfully - in terms of number of files and total size.

Any points to resolve this would be great help.

Thanks SV

SVen
asked 7 months ago217 views
2 Answers
2

Hi,

To get a summary of the objects within a bucket, run the following command:

aws s3 ls --summarize --human-readable --recursive s3://<bucket-name>/

This should give an output like this:

2021-10-07 21:32:57 452 Bytes foo/bar/car/petrol
2021-10-07 21:32:57 896 Bytes foo/bar/truck/diesel
2021-10-07 21:32:57 189 Bytes foo/bar/hybrid/battery
2021-10-07 21:32:57 398 Bytes vehicles.txt

Total Objects: 4
Total Size: 2.9 MiB

By comparing the number of files and total size from your on-premise file system and S3, you can verify if everything is transferred correctly.

References:
[1] https://aws.amazon.com/blogs/storage/find-out-the-size-of-your-amazon-s3-buckets/
[2] https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html

Thanks,
Atul

profile picture
answered 7 months ago
profile pictureAWS
EXPERT
reviewed 7 months ago
0

Hi,

Agreed with IBAtulAnand's answer for a quick check

A second level could be to compare a list of files by name and size from both sides (local and S3)

But, the most secure is to compute for each file a checksum (like MD5 or more advanced if you prefer) on both local and S3 and check if they are identical for a given file name. This will confirm to your client that files not only have same size individually but also that their content is identical for each individual bit. This very secure procedure will ensure that the transfer utility did not alter any data (while keeping it same size)

Best,

Didier

profile pictureAWS
EXPERT
answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions