s3 batch operations takes to long

0

Why s3 batch operation copy from one bucket to another in the same region takes a long time? I try to copy 8000 object with average size of 900 kb (using csv manifest with source bucket and s3 key, bucket doesn't have versioning) and job was in active status 69 hours and didn't complete so I just canceled the job.

Agan
질문됨 2년 전1635회 조회
1개 답변
1

Hi There

Take a look at this previous answer

https://repost.aws/questions/QUzH_mXBObTuO9iq_pvJkkoA/s-3-batch-operations-job-stays-active

A batch job performs a specified operation on every object that is included in its manifest. A manifest lists the objects that you want a batch job to process and it is stored as an object in a bucket. After you create a job, Amazon S3 processes the list of objects in the manifest and runs the specified operation against each object.

When a S3 batch job is "active" it means Amazon S3 is performing the requested operation on the objects listed in the manifest. While a job is Active, you can monitor its progress using the Amazon S3 console or the DescribeJob operation through the REST API, AWS CLI, or AWS SDKs.

The performance/speed of any particular Batch Operations job will depend on a variety of factors, including :

  • The number of objects in the manifest
  • The type of operation (Glacier restore, copy, Lambda function, etc)
  • The number of active jobs your account
  • The number of enqueued jobs for your account
  • Other traffic to the source and/or destination bucket
  • The size of the objects

You can refer to the following link for information on troubleshooting S3 batch jobs: https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-batch-operations/

If you need additional support, then you can also open a case with AWS Premium Support to review the details of this particular job.

profile pictureAWS
전문가
Matt-B
답변함 2년 전
  • Thank you for your response. So it's terns out that is's pretty useless tool? Cause I just test on 132 objects and it already 17 hours in active status

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠