By using AWS re:Post, you agree to the Terms of Use
/S3 sync handling of downloads beyond 36 hours or session token expiry/

S3 sync handling of downloads beyond 36 hours or session token expiry


When downloading large files (e.g. a few TB of data) via S3 CLI, the session token expires beyond 36 hours. I understand from here that it's not possible to extend or refresh temporary credentials.

I would like to know if s3 sync can resume the same downloading job with a new set of temporary credentials.

1 Answers

Yes with new credentials any object that has already been transferred won't be retransferred. However if one object is particularly large and will not complete within 36 hours even though the s3 sync commmand will use multipart upload you cannot resume from failed uploads in this scenario - see docs. You could run a multi part upload on large file which you can resume using the lower level cli commands - see here.

You might also want to consider using AWS Datasync if this is a process which you will be running regularly.

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions