s3 batch operations takes to long

0

Why s3 batch operation copy from one bucket to another in the same region takes a long time? I try to copy 8000 object with average size of 900 kb (using csv manifest with source bucket and s3 key, bucket doesn't have versioning) and job was in active status 69 hours and didn't complete so I just canceled the job.

Agan
asked 2 months ago50 views
1 Answer
1

Hi There

Take a look at this previous answer

https://repost.aws/questions/QUzH_mXBObTuO9iq_pvJkkoA/s-3-batch-operations-job-stays-active

A batch job performs a specified operation on every object that is included in its manifest. A manifest lists the objects that you want a batch job to process and it is stored as an object in a bucket. After you create a job, Amazon S3 processes the list of objects in the manifest and runs the specified operation against each object.

When a S3 batch job is "active" it means Amazon S3 is performing the requested operation on the objects listed in the manifest. While a job is Active, you can monitor its progress using the Amazon S3 console or the DescribeJob operation through the REST API, AWS CLI, or AWS SDKs.

The performance/speed of any particular Batch Operations job will depend on a variety of factors, including :

  • The number of objects in the manifest
  • The type of operation (Glacier restore, copy, Lambda function, etc)
  • The number of active jobs your account
  • The number of enqueued jobs for your account
  • Other traffic to the source and/or destination bucket
  • The size of the objects

You can refer to the following link for information on troubleshooting S3 batch jobs: https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-batch-operations/

If you need additional support, then you can also open a case with AWS Premium Support to review the details of this particular job.

profile picture
EXPERT
Matt-B
answered 2 months ago
  • Thank you for your response. So it's terns out that is's pretty useless tool? Cause I just test on 132 objects and it already 17 hours in active status

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions