AWSS3UploadTask - error completing upload Error: File size does not match between local file and file on s3

Question

Can anyone please explain to me possible causes of the following behavior?
I created an s3 bucket with amplify cli add storage. 
When I try to upload a file using `Storage.put` there is an error from title that file size doesn't  match.
I try to upload file manually from AWS s3 web interface dashboard, and the size on s3 is a bit smaller.

How can I investigate the reasons that the file is smaller on s3? 
when I download it back the size is the same with the original size in OS (Ubuntu)

I have checked this question with the same error, but in my case the behavior is same for image and other files. And I see the file size change even when I'am uploading directly to s3 dashboard

Answer

Hello,

Greetings for the day!

I understand that your are observing differences in object file size on your local system and on S3 bucket. The reason for this difference can be attributed towards additional header or metadata information which is required by each filesystem, this can also contain permission data such as who can read, write and execute a file along with filesystem block information.

This is also the reason why some operating systems have 2 sizes such as size and size on disk, and each operating system uses a different filesystem algorithm and protocol such as Ext3/Ext4/ReiserFS for Linux, NTFS for Windows and HFS/HFS+/, APFS for Mac. Amazon S3 does not work like a traditional filesystem with blocks and rather uses a key and prefix patterns for storage and all permission related information is controlled at the bucket level along with AWS IAM which is Identity and Access Management system. Hence, we observe this difference in object size.

If you wish to check the integrity of the file you can calculate the MD5 hash checksum of the Object before uploading and then again after downloading the uploaded object on the S3 bucket. More info here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

Thank you.

Answer

Hello folks,
Greetings & Happy New Year

I am currently facing the same issue.
I am trying to upload a .dmg file from my local laptop (MBP M2Pro, AFPS file system) to S3. The .dmg file is 396.91 Mb at source. The moment it is dragged and dropped in S3, it is reduced to ~380Mb and then the file is uploaded successfully. However, as a practice, I always download the file from S3 to test for any issues. This file is installed correctly but then shows that the installer file is corrupted.

Surprisingly, .apk file did exhibit similar pattern in file size reduction, but i was able to successfully install the .apk file from S3

ps: The checksum value of the file on my local machine and S3 are also different.

Not sure what's happening with this!
Any/ all help would be appreciated.

Thank You

AWSS3UploadTask - error completing upload Error: File size does not match between local file and file on s3

Relevant content