S3Transfer class ProgressPercentage writes frequently

0

I'm uploading a large file and I have it configured to use 1gb parts in the upload. I just recently added the ProgressPercentage class to S3Transfer and it's writing much more frequently than I expect. I would expect it to write once per part. Why is there 262k between values of what it's uploading? I'm using the standard class posted here, but I only modified how it writes.

EDIT: I modified my script to only print when it's an even percentage, so it will print at 0.00, 1.00, etc. Still, why is it doing this?

https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3/s3/transfer.html

2022-11-21 07:14:42,774 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3580887040 / 92093203968.0 (3.89%) 2022-11-21 07:14:42,852 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581149184 / 92093203968.0 (3.89%) 2022-11-21 07:14:42,930 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581411328 / 92093203968.0 (3.89%) 2022-11-21 07:14:42,972 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581673472 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,179 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581935616 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,240 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582197760 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,334 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582459904 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,428 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582722048 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,459 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582984192 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,475 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3583246336 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,662 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3583508480 / 92093203968.0 (3.89%)

asked a year ago206 views
2 Answers
0

Why wasn't this post shared with the Storage topic group?

answered a year ago
0

I see what's happening. The S3TransferConfig has a parm called io_chunksize that's set to 256 kb. I guess that would be configurable. I'll try that. I might leave it the way it is, though. Here's where I found that:

https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3/s3/transfer.html

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions