Multipart upload with aws S3 + checksums

0

I am trying to implement browser multipart upload to a S3 bucket. I should be able to pause and play the upload and also I'll like to automatically generate the checksums as I'm uploading.

I have tried several approaches and I've been hitting a wall. Some of the approaches I've tried.

  • Using the amplify S3 upload, this works well, but has the caveat that I can't generate the checksums automatically, to generate the checksums, I run a lambda function after file upload, the caveat is for large files, the lambda function times out. Also, I'll like to avoid going this route as I believe It's quite computationally expensive.
  • Using https://blog.logrocket.com/multipart-uploads-s3-node-js-react/. This is also similar to the above, the caveat is when I add the checksum algorithm to the upload part query, I get a checksum type mismatch occurred, expected checksum type sha256, actual checksum type: null site:stackoverflow.com s3. After a lot of googling, I'm not sure I can compute the checksums using presigned url.
  • and the current approach is to do away with the presigned url and send the chunked data to the lambda functions which then sends to the bucket. Since I'm managing everything with amplify, I run into some problems with API gateway(multipart/form-data). I have set the gateway to accept binary data and followed other fixes I found online but I’m stuck on execution failed due to configuration error unable to transform request.

How do I fix the above error and what will be the ideal approach to implement the functionalities(multipart file upload to support resumable uploads and checksum computation)

1 Answer
1

Based on your requirements, the ideal approach to implement the functionalities of multipart file upload to support resumable uploads and checksum computation would be to use the AWS SDK for JavaScript to upload the file in parts directly to S3, without using pre-signed URLs or Lambda functions. Also, you can use tools like ShareGate or Gs Richcopy 360 to directly and easily do this job.

Vima36
answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions