S3Transfer class ProgressPercentage writes frequently

0

I'm uploading a large file and I have it configured to use 1gb parts in the upload. I just recently added the ProgressPercentage class to S3Transfer and it's writing much more frequently than I expect. I would expect it to write once per part. Why is there 262k between values of what it's uploading? I'm using the standard class posted here, but I only modified how it writes.

EDIT: I modified my script to only print when it's an even percentage, so it will print at 0.00, 1.00, etc. Still, why is it doing this?

https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3/s3/transfer.html

2022-11-21 07:14:42,774 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3580887040 / 92093203968.0 (3.89%) 2022-11-21 07:14:42,852 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581149184 / 92093203968.0 (3.89%) 2022-11-21 07:14:42,930 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581411328 / 92093203968.0 (3.89%) 2022-11-21 07:14:42,972 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581673472 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,179 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581935616 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,240 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582197760 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,334 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582459904 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,428 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582722048 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,459 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582984192 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,475 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3583246336 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,662 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3583508480 / 92093203968.0 (3.89%)

gefragt vor einem Jahr221 Aufrufe
2 Antworten
0

Why wasn't this post shared with the Storage topic group?

beantwortet vor einem Jahr
0

I see what's happening. The S3TransferConfig has a parm called io_chunksize that's set to 256 kb. I guess that would be configurable. I'll try that. I might leave it the way it is, though. Here's where I found that:

https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3/s3/transfer.html

beantwortet vor einem Jahr

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen