s3 sync command is SLOW to start (On some data)

Hi MarqueeCReW,

Are you near from your S3 data? If no, consider enabling S3 Transfer Acceleration for faster uploads and downloads with S3. This can help improve the speed of transferring files over long distances.

The AWS CLI can perform operations in parallel to increase the speed of transferring files. You can adjust the max_concurrent_requests option to a higher value. Modify this in your AWS CLI configuration (~/.aws/config):

[default]
s3 =
    max_concurrent_requests = 20

You can adjust the number according to your bandwidth and system capabilities.

For large files, you can increase the multipart_chunksize to a larger value. Again, this can be adjusted in your AWS CLI configuration:

[default]
s3 =
    multipart_chunksize = 128MB

If you know specifically which files or types of files you want, you can use the --exclude and --include parameters to narrow down what's being synced. This minimizes the amount of data the sync command has to process.

For example, if you know you only want .txt files:

aws s3 sync s3://your-bucket/path/ local-path/ --exclude "*" --include "*.txt"

I hope this helps! If this solution works for you, please accept the answer. Otherwise, do leave a comment, and I'll try to assist you.

I understand you might have considered some basic checks, but it's worth reconfirming:

• Network Delays: There could be network-related delays originating from your end, your ISP, or any other intermediate nodes. Trying a different network or evaluating the current network's stability might provide insights.
• Local System Resources: Other active processes on your machine might be influencing the AWS CLI's performance. It's helpful to monitor the system's performance metrics, such as CPU and memory usage, during the s3 sync operation.
• Using --debug with AWS CLI: To get an in-depth view of what the CLI is doing, you can append the --debug option during your sync command. It'll yield extensive output, which can pinpoint where the potential slowdown might be.
• Alternative Tools: While AWS CLI is a preferred choice for many, various third-party tools offer syncing capabilities with S3. Exploring an alternative might help you determine if the delay is specifically related to the AWS CLI.