s3 sync command is SLOW to start (On some data)

Question

I daily run 2 CLI commands from my local PC.  One set of data has roughly 40 new files on s3 and the other has 1200+ files.

The smaller file set jumps into action, while i can stare at a black screen for quite a while until it starts downloading the first file.  I try to keep the s3 buckets clean (removing data so that there isn't so much to sync).

Is there something that I can do to make the sync start quicker?  Ultimately, I just want the downloads to complete faster.  I did think about COMPUTING the CLI command to make it static and download the current data, but because I'm not fully automated, I like the flexibility to download multiple days worth of data.

Vitor Castellani · Answer

I understand you might have considered some basic checks, but it's worth reconfirming:

* **Network Delays**: There could be network-related delays originating from your end, your ISP, or any other intermediate nodes. **Trying a different network** or evaluating the current network's stability might provide insights.
* **Local System Resources**: Other active processes on your machine might be influencing the AWS CLI's performance. It's helpful to **monitor the system's performance metrics**, such as CPU and memory usage, during the s3 sync operation.
* **Using --debug with AWS CLI**: To get an in-depth view of what the CLI is doing, you can append the --debug option during your sync command. It'll yield **extensive output, which can pinpoint where the potential slowdown might be**.
* **Alternative Tools**: While AWS CLI is a preferred choice for many, various third-party tools offer syncing capabilities with S3. Exploring an alternative might help you **determine if the delay is specifically related to the AWS CLI**.

Vitor Castellani · Answer

Hi MarqueeCReW,

Are you near from your S3 data? If no, consider enabling **S3 Transfer Acceleration** for faster uploads and downloads with S3. This can help improve the speed of transferring files over long distances.

The AWS CLI can perform operations in parallel to increase the speed of transferring files. You can adjust the **max_concurrent_requests** option to a higher value. Modify this in your AWS CLI configuration (~/.aws/config):

```
[default]
s3 =
    max_concurrent_requests = 20
```

You can adjust the number according to your bandwidth and system capabilities.

For large files, you can increase the **multipart_chunksize** to a larger value. Again, this can be adjusted in your AWS CLI configuration:

```
[default]
s3 =
    multipart_chunksize = 128MB
```

If you know specifically which files or types of files you want, you can use the **--exclude and --include parameters** to narrow down what's being synced. This minimizes the amount of data the sync command has to process.

For example, if you know you only want .txt files:

```
aws s3 sync s3://your-bucket/path/ local-path/ --exclude "*" --include "*.txt"
```

*I hope this helps! If this solution works for you, please accept the answer. Otherwise, do leave a comment, and I'll try to assist you.*

s3 sync command is SLOW to start (On some data)

Relevant content