S3 sync handling of downloads beyond 36 hours or session token expiry
When downloading large files (e.g. a few TB of data) via S3 CLI, the session token expires beyond 36 hours. I understand from here that it's not possible to extend or refresh temporary credentials.
I would like to know if s3 sync can resume the same downloading job with a new set of temporary credentials.
Yes with new credentials any object that has already been transferred won't be retransferred. However if one object is particularly large and will not complete within 36 hours even though the s3 sync commmand will use multipart upload you cannot resume from failed uploads in this scenario - see docs. You could run a multi part upload on large file which you can resume using the lower level cli commands - see here.
You might also want to consider using AWS Datasync if this is a process which you will be running regularly.
Relevant questions
S3 sync handling of downloads beyond 36 hours or session token expiry
asked a month agoMigrate on-prem data( files) to S3 using Snowball and data sync using DataSync
asked 4 months agoS3-S3 Datasync Fails with error: The endpoint has already opened a volume for this session
asked 6 months agoCost difference between downloading S3 files into Lambda function via HTTPS VS SDK?
asked 5 months agoS3 mv and rename does not work for large files
Accepted Answerasked 5 years agoS3 sync command is using 90% of CPU of the server
asked 3 years agoS3 for backup of large small files on-premise
Accepted Answerasked 2 years agoOptions to accelerate s3 copy of 4TB worth files between S3 buckets in same region
Accepted Answerasked 2 years agoDrastically increase data size after sync local data to s3 deep archive
asked 3 years agoSNOWBALL Export transfer progress has been stopped going on 36 hours!
asked 3 years ago