S3Transfer class ProgressPercentage writes frequently

0

I'm uploading a large file and I have it configured to use 1gb parts in the upload. I just recently added the ProgressPercentage class to S3Transfer and it's writing much more frequently than I expect. I would expect it to write once per part. Why is there 262k between values of what it's uploading? I'm using the standard class posted here, but I only modified how it writes.

EDIT: I modified my script to only print when it's an even percentage, so it will print at 0.00, 1.00, etc. Still, why is it doing this?

https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3/s3/transfer.html

2022-11-21 07:14:42,774 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3580887040 / 92093203968.0 (3.89%) 2022-11-21 07:14:42,852 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581149184 / 92093203968.0 (3.89%) 2022-11-21 07:14:42,930 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581411328 / 92093203968.0 (3.89%) 2022-11-21 07:14:42,972 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581673472 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,179 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3581935616 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,240 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582197760 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,334 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582459904 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,428 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582722048 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,459 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3582984192 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,475 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3583246336 / 92093203968.0 (3.89%) 2022-11-21 07:14:43,662 - aws_pc_backup - [CRITICAL] - Progress for C:\Temp_Download\C_VOL-b031.spf 3583508480 / 92093203968.0 (3.89%)

질문됨 일 년 전216회 조회
2개 답변
0

Why wasn't this post shared with the Storage topic group?

답변함 일 년 전
0

I see what's happening. The S3TransferConfig has a parm called io_chunksize that's set to 256 kb. I guess that would be configurable. I'll try that. I might leave it the way it is, though. Here's where I found that:

https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3/s3/transfer.html

답변함 일 년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠