Network issue to S3: Timeouts in S3?

Hey Philipp, based on the information provided, it's likely related to idle connection timeouts.

Idle Connection Timeouts

VPC Endpoints, including those for S3, have a fixed idle timeout of 350 seconds. After this period of inactivity, the connection is terminated, which can lead to the "SocketException: Connection reset" error you're experiencing.

We can try a few things:

1. Use TCP Keepalive:

   • Implement TCP Keepalive in your Java application to send periodic packets and keep the connection alive.
   • By default, Linux-based systems have TCP Keepalive configured to trigger after 7200 seconds (2 hours), which is likely why you're seeing the issue after about an hour.

2. Configure TCP Keepalive:

   • Adjust the TCP Keepalive settings in your operating system or Java application to send probes more frequently.
   • For example, you could set:
     • tcp_keepalive_time: 300 seconds (5 minutes)
     • tcp_keepalive_intvl: 60 seconds
     • tcp_keepalive_probes: 5

3. Implement Application-Level Keepalive:

   • If TCP Keepalive is not sufficient, implement an application-level keepalive mechanism.
   • Periodically send small requests to S3 (e.g., a HEAD request on an object) to keep the connection active, this might trigger additional API costs depending on the number of requests you make.

4. Handle Connection Resets:

   • Implement retry logic in your application to handle connection resets gracefully.
   • When a reset occurs, re-establish the connection and resume the operation.

5. Use AWS SDK?:

   • If not already doing so, use the AWS SDK to interact with S3.
   • The SDK provides built-in retry mechanisms and can handle many connection issues automatically.

Here's an example in Java:

• Set appropriate timeouts in the AWS SDK:

  S3Client s3Client = S3Client.builder()
      .overrideConfiguration(b -> b
          .apiCallTimeout(Duration.ofSeconds(300))
          .apiCallAttemptTimeout(Duration.ofSeconds(60)))
      .build();

Here's a few links to help you while chasing the real culprit:

https://docs.aws.amazon.com/codeguru/detector-library/java/avoid-reset-exception-rule/ and https://aws.amazon.com/blogs/networking-and-content-delivery/implementing-long-running-tcp-connections-within-vpc-networking/

Hi,

The workaround mentioned above will be helpful with connection reset issue if there was already a TCP connection created with S3 and the existing connection was closed. However if there is no existing connection and if this error "connection reset" is seen for a new connection this could be caused due to many reasons listed in this AWS blog 'How do I troubleshoot ConnectionReset errors when I upload or download objects from Amazon S3?' : https://repost.aws/knowledge-center/s3-connect-reset-error

If this solutions do not resolve the issue, please feel free to reach out to AWS premium support to further troubleshoot the issue.

Thanks,

To troubleshoot the "Connection reset" error when reading from or writing to S3, you can follow these steps:

Test connectivity by running commands to check if your machine can establish a connection to S3 over HTTP or HTTPS. You can bypass SSL validation using the --no-verify-ssl parameter to isolate issues related to SSL validation.

Analyze network traffic by taking a packet capture to determine if the client or server throws the RST flag during processes like DNS lookup, TCP handshake, SSL handshake, and data transfer. If the RST flag is thrown during SSL handshake, ensure your machine trusts the certificate returned by Amazon.

Debug the connection by running the AWS CLI cp command with the --debug flag and analyze the request headers, request method, and response. Check for proxy errors or proxy connection failures in the output, which may indicate issues with an intermediary resource like a NAT gateway, firewall, or load balancer.

Check if TCP Keep-Alive is turned on by running the following command: sysctl -a | grep tcp_keepalive ( refer the provided detailed explanation by diegogmAtAWS about Keepalive in this article )

Increase the fs.s3.maxConnections value to accommodate the concurrency of your application. However, this may be a temporary fix, and you should ensure that S3Objects connections are properly closed in your code using the S3Object.close() method or a try-with-resources block.

https://repost.aws/knowledge-center/s3-connect-reset-error