CodeBuild S3 upload performance inconsistency: Identical machines showing 10min vs 4min upload times

The inconsistent upload speeds you're experiencing between identical CodeBuild machines could be caused by several factors, even when the machines appear to have the same configuration.

Here are some potential causes and solutions:

1. Network Conditions: Even within the same AWS region, network paths can differ between instances, leading to varying performance. The machines might be in different availability zones or using different network paths to reach S3. Check the network configurations of your CodeBuild project and compare them for builds with different upload speed. https://docs.aws.amazon.com/codebuild/latest/userguide/vpc-support.html

2. Parallelization Settings: The AWS CLI has default settings for S3 transfers that might not be optimized for your use case. Try configuring the following parameters:

• Increase concurrent requests: aws configure set default.s3.max_concurrent_requests 16
• Increase queue size: aws configure set default.s3.max_queue_size 10000
• Set multipart threshold: aws configure set default.s3.multipart_threshold 64MB
• Set chunk size: aws configure set default.s3.multipart_chunksize 64MB
• Consider disabling payload signing: aws configure set default.s3.payload_signing_enabled false

3. Resource Contention: One machine might be experiencing higher CPU usage or disk I/O contention than the other, which can affect upload speeds. Check if there are other processes running on the slower machine. You can check the resource utilization of individual builds using the documentation here - https://docs.aws.amazon.com/codebuild/latest/userguide/monitoring-utilization-metrics.html

4. S3 Transfer Acceleration: Consider enabling S3 Transfer Acceleration, which can optimize long-distance transfers between clients and S3 buckets.

5. File Characteristics: If your build produces many small files, the overhead of individual requests can slow down the process. Consider bundling files together if possible.

6. Multipart Upload Issues: For larger files, ensure multipart uploads are properly configured. Sometimes incomplete multipart uploads can cause hangs or slowdowns.

7. Debug Mode: Run your uploads with the --debug flag to get more detailed information about what might be causing the slowdown.

If these suggestions don't resolve the issue, you might want to try increasing the max_concurrent_requests further (to 64 or even 100) if your system can handle it. However, be cautious as setting this too high could overwhelm the system resources and actually decrease performance.

Overall, Inconsistent upload/download speeds have multiple contributing factors. Please refer this document for more details.

Sources
Upload to AWS S3 is pathetically slow | AWS re:Post
I am getting very low speed while uploading data to AWS S3 deep archive storage class | AWS re:Post
Why do S3 CLI uploads sometimes hang mid-transfer with no error? | AWS re:Post
S3 Transfer Rates Capped at 2.8mb/s ??? How Can I Speed This Up? | AWS re:Post