S3 sync limitations on index size?

Question

I have a small number of files that I will be syncing to S3 (generational files daily).  On the local side I will be running the sync with only about 36 files daily.  However, over time the stored files in S3 will grow significantly and I want to be sure I will not encounter an issue if there are thousands of files in S3 near the end of the year.

In other words, when sync is determining whether the 36 files I am syncing are new or not - will the amount already stored in S3 cause any issue when it is comparing last modified data, etc.?  Will the index be too large to effectively sync my small batch of daily files?

I hope this makes sense.

Answer

Your question is about the repetitive task each day to sync it?

The service where you are using is correct to synchronize any local data to S3 bucket.

Be aware to have the latest version of CLI: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-troubleshooting.html

I'm 100% sure you can upload small files into s3 bucket without problem.  (Max file size 5TB)

Answer

Hi there,

When running the "aws s3 sync" command, the operation lists out the contents of the source and destination before determining the files that need to be copied to the destination. If a location is S3, the CLI uses ListObjectsV2 requests to list the contents of the bucket. Each API call returns up to 1000 entries, and so a few thousand files will result in only a handful of LIST requests, which should be fine performance-wise. If you continue to scale and you begin to see any performance issues in the time it takes to complete the process, I might recommend trying to split up the overall task to sync smaller subdirectories to limit the number of objects involved on either end. From personal experience, I've seen sync be used with millions of objects without issues

S3 sync limitations on index size?

Relevant content