What is the max object-per-second throughput possible when copying S3 objects from Standard to Glacier Instant Retrieval with S3 Batch Operations?

0

As in the title: what is the max object-per-second throughput possible when copying objects from Standard to Glacier Instant Retrieval with S3 Batch Operations?

In our dev/sandbox environments we have created jobs to copy objects between storage classes using a manifest (pre-computed .csv.gz, not using manifest generator). So far we have only seen ~500 objects-per-second throughput in dev but in prod we are looking to move ~10 billion small objects (<1MiB) so we need it to go faster.

I'm aware that we can split the job up and that is something we will explore but how can we boost throughput per job? Is there any way we can get the speed up to more like ~10k objects-per-second? Does the throughput depend on the target storage class (e.g. Intelligent Tiering versus GIR)? Will it be faster or slower if the bucket has more "partitions" allocated?

Thanks in advance, James

  • Few questions:

    1. Are you copying objects within the same bucket or to a different bucket?
    2. If you doing it within same bucket, have you considered Lifecycle?
    3. How are your objects named, are you using prefixes, if so what is your naming convention, if any?
  • Thanks @Tom-B,

    1. same bucket (with a reduced non-current version expiration to avoid double-paying for storage)
    2. we looked at lifecycle policies but have >1,000 prefixes we want to move
    3. the objects are spread out by using a hash (sha256+base64) at the start of the key

    PS with further testing in the dev environment we're now seeing ~3,000 tps.

James
asked 13 days ago397 views
1 Answer
0
Accepted Answer

Your requests per second will depend on how you've organised your data into prefixes. You can find out more about the performance here and considerations around prefixes.

Also this YouTube Video from Re:Invent 2023 goes deeper into this and is definitely worth a watch: https://youtu.be/sYDJYqvNeXU?list=PL2yQDdvlhXf83bp752n992F52HWaR_js3&t=967

AWS
Tom-B
answered 12 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions