What are best practices for downloading large files using C++ aws::crt::http::httpclient and http GET

Question

We are developing on a linux device using C++ and aws::crt.  We have file downloads working using http GET, and have successfully downloaded files of around 100 MB.  The function is fairly basic, using the default eventLoopGroup and defaultHostResolver and default TlsContextOptions.  The GET requests are to signed URLs provided as part of IoT jobs, so the http client connections are created only when a job is active and only as long as needed to download the file.

The concepts that I'm not sure apply to this application are backpressure and connection interruptions.  Backpressure seems to only be relevant for streaming applications, and not for http GET requests that can return files on the order of 100 MB.  Also, if backpressure is to be implemented, the server responding to the GET request and sending the large file would have to manage this, and I'm not sure the AWS servers that download files to devices as part of IoT jobs include this.

If for some reason the download is interrupted due to a loss of network connection, for example, would this lead to a timeout of some sort, or will the thread hosting the download function hang indefinitely?  I don't see timeouts mentioned in the HttpStream documentation.  SocketOptions ConnectTimeoutMs is set to 5 seconds, but it's my understanding that this just applies to establishing the initial connection to the server.  If an application is simply establishing a connection, immediately downloading a file and disconnecting, I don't think keep alive is relevant.

With regard to handling connection interruptions, ChannelBootstrap can be assigned a handler for connection interruption, but we do not use ChannelBootstrap in the current implementation, and I'm not clear on whether it would be necessary to include this for what would be considered a robust implementation.

The current implementation is working fine, and failure will lead to a later retry; just looking for guidance on any additional error handling that would be appropriate.

re:Post Agent · Answer

When downloading large files using C++ aws::crt::http::httpclient and HTTP GET, there are several best practices to consider for a robust implementation:

1. Use byte-range requests: Instead of downloading the entire file in one request, split it into multiple smaller requests. This approach is particularly useful for large files (e.g., 60GB or more) as it can help maintain consistent bandwidth and improve retry capabilities if interruptions occur. Typically, byte-range sizes of 8-16 MB are recommended.

2. Implement parallel downloads: Use concurrent requests to download different byte ranges of the file simultaneously. This can significantly increase throughput. As a general guideline, make one concurrent request for each 85-90 MB/s of desired network throughput.

3. Handle timeouts and retries: While the SocketOptions ConnectTimeoutMs handles the initial connection, you should implement additional timeout mechanisms for the download process itself. This prevents the thread from hanging indefinitely if there's a network interruption. Implement a retry mechanism with exponential backoff for failed requests.

4. Error handling: Implement robust error handling to manage various scenarios such as network interruptions, HTTP errors, or unexpected responses. This includes handling partial downloads and resuming from where the download was interrupted.

5. Monitor and adjust performance: Start with a single request and gradually increase the number of concurrent requests while monitoring performance. Adjust based on your specific network conditions and requirements.

6. Consider using Transfer Acceleration: For geographically distant transfers, Amazon S3 Transfer Acceleration can help minimize latency and improve download speeds.

7. Implement checksum verification: After downloading, verify the integrity of the file using checksums to ensure the download was successful and complete.

While backpressure is more relevant for streaming applications, implementing a form of flow control in your application can help manage resource usage, especially when dealing with very large files or multiple concurrent downloads.

Regarding connection interruptions, while you're not using ChannelBootstrap, it's still important to implement a mechanism to detect and handle disconnections. This could involve setting read timeouts on the HTTP client and implementing a reconnection strategy.

Even though your current implementation is working fine for 100 MB files, these practices will make your system more robust and scalable, especially when dealing with larger files or less stable network conditions.  
**Sources**  
[Horizontal Scaling and Request Parallelization for High Throughput - Best Practices Design Patterns: Optimizing Amazon S3 Performance](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/horizontal-scaling-and-request-parallelization-for-high-throughput.html)  
[Use Byte-Range Fetches - Best Practices Design Patterns: Optimizing Amazon S3 Performance](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html)  
[Download bandwidth | AWS re:Post](https://repost.aws/questions/QUR6lcygtoRQyqmLlgosQYgg/download-bandwidth)  
[Performance design patterns for Amazon S3 - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html)

What are best practices for downloading large files using C++ aws::crt::http::httpclient and http GET

Add your answer

Relevant content