As in the title: what is the max object-per-second throughput possible when copying objects from Standard to Glacier Instant Retrieval with S3 Batch Operations?
In our dev/sandbox environments we have created jobs to copy objects between storage classes using a manifest (pre-computed .csv.gz, not using manifest generator). So far we have only seen ~500 objects-per-second throughput in dev but in prod we are looking to move ~10 billion small objects (<1MiB) so we need it to go faster.
I'm aware that we can split the job up and that is something we will explore but how can we boost throughput per job? Is there any way we can get the speed up to more like ~10k objects-per-second? Does the throughput depend on the target storage class (e.g. Intelligent Tiering versus GIR)? Will it be faster or slower if the bucket has more "partitions" allocated?
Thanks in advance,
James
Few questions:
Thanks @Tom-B,
PS with further testing in the dev environment we're now seeing ~3,000 tps.