What is the max object-per-second throughput possible when copying S3 objects from Standard to Glacier Instant Retrieval with S3 Batch Operations?

0

As in the title: what is the max object-per-second throughput possible when copying objects from Standard to Glacier Instant Retrieval with S3 Batch Operations?

In our dev/sandbox environments we have created jobs to copy objects between storage classes using a manifest (pre-computed .csv.gz, not using manifest generator). So far we have only seen ~500 objects-per-second throughput in dev but in prod we are looking to move ~10 billion small objects (<1MiB) so we need it to go faster.

I'm aware that we can split the job up and that is something we will explore but how can we boost throughput per job? Is there any way we can get the speed up to more like ~10k objects-per-second? Does the throughput depend on the target storage class (e.g. Intelligent Tiering versus GIR)? Will it be faster or slower if the bucket has more "partitions" allocated?

Thanks in advance, James

  • Few questions:

    1. Are you copying objects within the same bucket or to a different bucket?
    2. If you doing it within same bucket, have you considered Lifecycle?
    3. How are your objects named, are you using prefixes, if so what is your naming convention, if any?
  • Thanks @Tom-B,

    1. same bucket (with a reduced non-current version expiration to avoid double-paying for storage)
    2. we looked at lifecycle policies but have >1,000 prefixes we want to move
    3. the objects are spread out by using a hash (sha256+base64) at the start of the key

    PS with further testing in the dev environment we're now seeing ~3,000 tps.

James
質問済み 1ヶ月前412ビュー
1回答
0
承認された回答

Your requests per second will depend on how you've organised your data into prefixes. You can find out more about the performance here and considerations around prefixes.

Also this YouTube Video from Re:Invent 2023 goes deeper into this and is definitely worth a watch: https://youtu.be/sYDJYqvNeXU?list=PL2yQDdvlhXf83bp752n992F52HWaR_js3&t=967

AWS
Tom-B
回答済み 1ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ