Can't Download Large Files on AWS Batch Job, Max Retries Exceeded Error

Question

I'm experiencing failures when downloading large video files (>4GB) in my AWS Batch job for video encoding. The download works fine for smaller files, but consistently fails for files exceeding 4GB. The job runs on an** r4.8xlarge** instance with sufficient CPU, memory, and disk space (26.54 GB available). I'm getting "Max Retries Exceeded". ***Note that, it works locally fine and the file is downloaded, but not on Aws Batch Job***

```
           # Enhanced configuration with longer timeouts for AWS Batch
            config = Config(
                signature_version='s3v4',
                retries={'max_attempts': 3, 'mode': 'standard'},
                connect_timeout=5,
                read_timeout=60,
            )

r2 = boto3.client(
                's3',
                endpoint_url=CLOUDFLARE_R2_PUBLIC_URL,
                aws_access_key_id=CLOUDFLARE_R2_ACCESS_KEY,
                aws_secret_access_key=CLOUDFLARE_R2_SECRET_KEY,
                region_name='auto',
                config=config
            )
```
` r2_client.download_file(CLOUDFLARE_R2_BUCKET, key, local_path)
`

TheQuietBuilder · Accepted Answer

This sounds like a timeout issue related to how long the connection stays open during large downloads. Even with increased read_timeout, AWS Batch jobs may still hit retries if there’s network variability or if the connection is throttled.

You might try increasing both read_timeout and connect_timeout further, or use TransferConfig with multipart download.

Also, double-check if the batch job role has limits on outbound bandwidth.

re:Post Agent · Answer

When downloading large files (>4GB) in AWS Batch jobs, several factors could be causing your "Max Retries Exceeded" error:

1. Timeout settings: Your current configuration has a read_timeout of 60 seconds, which is likely insufficient for downloading files larger than 4GB. For large files, consider increasing this value significantly (to several minutes) to accommodate the longer download time.

2. Network throughput: Large file transfers can be impacted by network constraints. The r4.8xlarge instance should have good network performance, but there could still be bandwidth limitations when transferring from Cloudflare R2.

3. Connection stability: When downloading large files, the connection needs to remain stable for a longer period. Any network fluctuation can cause the download to fail.

4. Chunking approach: For files larger than 4GB, consider implementing a chunking strategy where you download the file in smaller segments and then reassemble them. This can help manage memory usage and reduce the impact of potential failures.

5. Retry strategy: Your current configuration only attempts 3 retries. For large files, you might want to increase this and implement an exponential backoff strategy.

Try modifying your configuration with these settings:
```
config = Config(
signature_version='s3v4',
retries={'max_attempts': 10, 'mode': 'adaptive'},
connect_timeout=30,
read_timeout=600,  # 10 minutes
)
```

Additionally, you could implement a multipart download approach using the boto3 S3 transfer manager, which is designed to handle large file transfers more reliably by breaking them into smaller chunks.  
**Sources**  
[AWS Batch job, FFmpeg video encoding issues. | AWS re:Post](https://repost.aws/questions/QUwSPWT8zJTw2_tl0oVHn7xA/aws-batch-job-ffmpeg-video-encoding-issues)  
[Downloading multiple images | AWS re:Post](https://repost.aws/questions/QUU3vLIPAlSjyYBdt6OZfnbQ/downloading-multiple-images)  
[Accessing S3 Objects takes time | AWS re:Post](https://repost.aws/questions/QUFhvIJzVwS76Y6Fi1EzoZ_w/accessing-s3-objects-takes-time)

Can't Download Large Files on AWS Batch Job, Max Retries Exceeded Error

Relevant content