com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool"

Question

I encountered this error when writing a large size Parquet file onto an S3 bucket, "com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool"
I included the following, but still the job stops.
[
    {
      "Properties": {
        "fs.s3.maxConnections": "500",
      }
    }
 ]
When I reduce the size of the data, everything works fine. 
Any insights would be very much appreciated. Thanks

Answer

Hi rePost-User-9456311,
     
    From my understanding, this seems like this is an issue about the parquet file size since when you reduce the size of the data everything works. I would try to perform a multipart upload with a SDK and see if that solves your issue or not. 
     
    Here are some links that should help
    https://aws.amazon.com/premiumsupport/knowledge-center/s3-large-file-uploads/
    https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool"

Relevant content