Multipart Upload > 900 seconds


What is the best practice, using JS SDK v3, to perform a multipart upload of a very large file that takes longer than 15 minutes to complete? For example, I can imagine a scenario in which 10000 5GB parts can't be uploaded in under 900 seconds.

A couple questions which may illustrate my comprehension level:

  1. Is the 900 second timeout enforced between the creation of the upload id and the UploadPart execution? That is my interpretation of the errors I'm getting. Or, is the timeout between UploadPartCommand object creation and execution?
  2. Is it possible to use multiple UploadIds to send all the parts of a very large file?
  3. If (2) is affirmative, is it correct to assume only one upload id should be used to "complete" the multipart upload and all others should be aborted, if not, how are multiple upload id's handled in order to stitch together parts of multiple uploads into a single object?
  4. Am I even in the right ballpark here, i.e., does everything need to complete within 900 seconds? This excerpt from the userguide suggests otherwise: "Pause and resume object uploads - You can upload object parts over time. After you initiate a multipart upload, there is no expiry; you must explicitly complete or stop the multipart upload."


asked a year ago60 views
1 Answer

I've resolved the issue as I suggested previously.

  1. Platoon or stagger upload requests into arbitrary batch sizes and execute batches in sequence
  2. Wait for all current batch responses before creating the subsequent batch
  3. Repeat for all batches

I've requested a documentation modification to indicate the following:

  1. A single UploadPartCommand must be received by the server within 15 minutes (900 seconds) of its creation
  2. A single UploadPartCommand that is not received by the server within 15 minutes of its creation will fail with a misleading "RequestTimeTooSkewed" error.
  3. "RequestTimeTooSkewed" error content, and documentation, and customer support all imply or explicitly state that the error is caused by incorrect client system time. In fact, the problem in this case is that the UploadPartCommand was merely created 15 minutes or more before it was received by the server.

This (#3) can happen as a matter of course when processing very large files. It requires only seconds to generate thousands of UploadPartCommand objects, each of will will have effectively the same creation time. Depending on bandwidth, part size, and part count, it can easily take more than 15 minutes to process these requests. Simultaneous UploadPartCommand requests must be platooned into manageable batches that can be entirely resolved within 15 minutes, lest subsequent, simultaneously created and queued requests fail.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions