- Newest
- Most votes
- Most comments
The slow S3 upload speeds you're experiencing are not typical for Amazon S3, and there are several ways to improve your transfer rates. S3 is designed to handle high-volume data transfers efficiently, and it's capable of much higher speeds than what you're currently seeing.
Here are some recommendations to improve your upload speeds:
-
Use Amazon S3 Transfer Acceleration: This feature optimizes long-distance transfers between clients and S3 buckets by using AWS edge locations to route data more efficiently.
-
Implement multipart uploads: For large files, using multipart uploads can dramatically increase transfer speeds by uploading parts of a file in parallel.
-
Optimize AWS CLI settings: If you're using the AWS CLI, you can configure it with the following settings to improve performance:
- Increase concurrent requests:
aws configure set default.s3.max_concurrent_requests 16 - Increase queue size:
aws configure set default.s3.max_queue_size 10000 - Set multipart threshold:
aws configure set default.s3.multipart_threshold 64MB - Set chunk size:
aws configure set default.s3.multipart_chunksize 64MB - Consider disabling payload signing:
aws configure set default.s3.payload_signing_enabled false
-
Check your network connection: Ensure you're using a fast, stable internet connection. Wired connections are generally more reliable than wireless for large data transfers.
-
Consider your geographic location: If you're far from the S3 region you're uploading to, you might experience slower speeds. Using S3 Transfer Acceleration or choosing a closer region can help.
-
Check for client-side limitations: Ensure your client machine isn't the bottleneck. Factors like CPU usage, disk I/O, or other running processes could limit your upload speed.
If these suggestions don't improve your upload speeds significantly, you might want to try increasing the concurrent requests further (to 64 or even 100) if your system can handle it. If performance still remains unchanged, the bottleneck might be your internet connection's upstream bandwidth.
S3 is capable of handling very high transfer rates, often reaching hundreds of megabits per second or even multiple gigabits per second in optimal conditions.
Sources
S3 Transfer Rates Capped at 2.8mb/s ??? How Can I Speed This Up? | AWS re:Post
I am getting very low speed while uploading data to AWS S3 deep archive storage class | AWS re:Post
Relevant content
- asked 2 years ago
