Can I retrieve Custom Metadata from an incomplete multipart upload?

0

If yes, how?

Context: I am implementing multipart upload with resuming ability. After an upload interruption, the user re-selects the same file from his drive (or maybe from another machine). I need to match that file against one of the existing incomplete UploadIds, if there is one for that file. Goal is to re-use that UploadId to resume the upload. How can I do this if the end-client has no idea of the UploadId?

I cannot fully rely on file names. It is possible that the user has changed the file name after the interruption, or that he has another file with the same file name (collision).

ListMultipartUploads gives me all the UploadId and the Key of incomplete uploads. But not any custom metadata that I have defined when using CreateMultipartUpload. I tried to to a headObject request to the Key of the upload to get the custom metadata, but I get a 404 (understandably).

Ideally I would do some sort of HeadObject request against the UploadId in order to get the custom metadata, in which I would have stored information that helps me identify the file (say a file checksum, or file checksum + user hash).

I don't think I can use ETags, since S3 calculates ETags as a checksum of the parts checksums, so S3 has no idea of the final ETag until the end, and cannot give it back to me when enquiering about the incomplete UploadId.

So the only option I can see is:

  • to use the Key as a match. I would need to name the file as a predictible checksum of the whole file itself, and name it this way. It feels wrong. Custom metadata feel more like the place for this.
  • maintain somewhere a key-value records matching my checksum and UploadId. Make sure to delete them when UploadId are aborted or completed... more complicated, also when using aborting rules with the bucket lifecycle

Am I doing this wrong, any ideas? Any way to access custom metadata form incomplete multiparts uploads? The data is somewhere, since the custom metadata is applied to the final object after the final CompleteMultipartUpload request is processed.

Thank you!

질문됨 4달 전231회 조회
1개 답변
0

This is definitely an interesting question that I haven't seen yet before. I don't personally know a great answer, but did a quick search and it appears that boto3 may have the ability to tag the metadata in parts. I also came across an SDK doc for customizing multi-parts (PHP).

I hope this steers you towards the right direction and good luck!

P.S. Don't forget to setup a Lifecycle rule to delete off those incomplete multi-part uploads.

profile pictureAWS
전문가
David
답변함 4달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠