Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

AWS S3 UploadPartCommand is returning a checksum for FULL_OBJECT checksum check

0

I am using @aws-sdk/s3-client to upload a file in multiple part. The thing is that upload part is generating a checksum for each UploadPartCommand. I suspect that I am doing something wrong, but my console.logs are not saying anything useful. You can see here how you can reproduce it in your system.

Change the env variables to the ones you get from the AWS and remove the OBJECT_STORAGE_ENDPOINT. It should be used only for local testing with Minio. It says something like the calculated CRC32 does not match the one specified in the CompleteMultipartUploadCommand.

  • Here is where you can see the UploadPartCommand is returning a checksum for each part.
  • While you can clearly see that I have said please use a FULL_OBJECT checksum check here. BTW feel free to add more console.logs to see if I am setting those configs correctly.
  • And here is where I try to complete the file upload.
  • Here is my e2e test.
  • BTW here is the simplified version which works, but I am doing the exact same thing in my NestJS app and IDK why it is failing.

Any help?

  • kasir-barati
    2 years ago

    So here is what should happen:

    1. You create a new multipart upload (CreateMultipartUploadCommand).
      • Here you can say I need COMPOSITE or FULL_OBJECT checksum check.
      • In both cases it will expect to receive the algorithm (in FULL_OBJECT we can only use CRC base algorithms).
    2. You start uploading them (UploadPartCommand).
      • You should not send an algorithm in this step if you wanna use FULL_OBJECT checksum check.
      • Just send the data and part number.
    3. Tell AWS S3 that you're finished (CompleteMultipartUploadCommand).
      • Now you need to send the checksum of the whole object (not each chunk) to the AWS so that they can calculate it on their servers and compare the two.
      • It will reject in case of mismatch.

    And this is what is happening right now:

    AWS S3 is generating a checksum for each part (COMPOSITE) and it is ignoring my setup I've sent it in CreateMultipartUploadCommand.

    Ref.

Topics
Storage
Tags
Amazon Simple Storage Service
Language
English

asked 2 years ago1.9K views

2 Answers
0
Accepted Answer

I was wrong, actually we need to keep those returned checksums for each part and send it back in the complete multipart command: As I have explained it here. You can find a working example written in NodeJS/NestJS here: https://dev.to/kasir-barati/nestjs-aws-s3-multipart-file-upload-grpc-e2e-testing-550e

answered 2 years ago

-2

Based on your description and the code you've shared, it appears that you're encountering an issue with the checksum validation during a multipart upload to Amazon S3. Let's break down the problem and provide some insights:

  1. Checksum for each part: It's normal for the UploadPartCommand to return a checksum for each part. This is part of S3's data integrity mechanism for multipart uploads. Each part is individually checksummed to ensure its integrity during transmission.

  2. FULL_OBJECT checksum: You're correctly specifying the FULL_OBJECT checksum type in the CreateMultipartUploadCommand. This is used for the final validation of the entire object after all parts have been uploaded.

  3. Checksum mismatch error: The error you're seeing suggests that the calculated CRC32 checksum for the entire object doesn't match the one you're providing in the CompleteMultipartUploadCommand.

Here are some potential issues and solutions:

  1. Ensure that you're calculating the full object checksum correctly before starting the upload. This should be done on the entire file, not just the individual parts.

  2. Make sure you're passing the correct checksum to the CompleteMultipartUploadCommand. The checksum should be base64 encoded.

  3. Verify that all parts are being uploaded correctly and in the right order. Any missing or corrupted part could cause a checksum mismatch.

  4. Double-check that you're using the same checksum algorithm (CRC32 in this case) consistently throughout the process.

  5. If you're pre-computing the checksum, ensure it's done on the exact same data that's being uploaded. Any discrepancy in the data will result in a different checksum.

  6. Consider using a different checksum algorithm like CRC32C or SHA256 to see if the issue persists.

  7. Make sure you're using the latest version of the AWS SDK, as there have been improvements to checksum handling in recent versions.

If the simplified version works but the NestJS app doesn't, there might be some subtle differences in how the data is being handled or how the SDK is being configured in the NestJS environment. You may want to add more detailed logging in your NestJS app to compare the exact values being used at each step with those in your simplified version.

Remember, when using FULL_OBJECT checksum type, you need to provide the checksum for the entire object in the CompleteMultipartUploadCommand, not the individual part checksums. If you're unsure about the checksum calculation, you could try removing the checksum parameter from the CompleteMultipartUploadCommand as a test to see if the upload completes successfully without it.
Sources
UploadPart - Amazon Simple Storage Service
Introducing default data integrity protections for new objects in Amazon S3 | AWS News Blog
UploadPartResponse (AWS SDK for Java - 2.21.29)

answered 2 years ago

  • kasir-barati
    2 years ago

    But the main point is that the UploadPartCommnad should not return a checksum when you say you need FULL_OBJECT checksum check. You AI needs a lot of work, it lacks what we call it common sense.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content