Skip to content

Intemittent SES connection timeouts from elastic beanstalk Tomcat app, DNS caching?

0

I have a Tomcat app running in EC2 instances created by Elastic Beanstalk. The application uses SES to send emails using the SesClient class in the AWS Java SDK. Most of the time, this works fine. However, intermittently, sending emails fails due to a connection timeout, for example [email.eu-west-1.amazonaws.com/34.252.104.46] failed: Connect timed out (SDK Attempt Count: 4)

I have disabled the DNS cache in the JVM by calling java.security.Security.setProperty("networkaddress.cache.ttl" , "0") at startup. I have set Cache=no in /etc/systemd/resolved.conf using a dns-cache.config file in .ebextensions. I even added some trace to resolve the hostname to an IP address every 10s to see what was going on. The resolved IP address does not match the one periodically resolved from my local machine, with the EC2 instance often lagging behind.

Is anyone able to help further, as I am tearing my hair out here.

Thanks!

2 Answers
0

You could try switching from the public endpoint into a privatelink one instead as SES is supported: https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-privatelink-support.html but this will a: likely increase costs and b: will definitely increase complexity of the architecture as route53, and privatelink endpoints are added into the mix. but.. it may help. the key change once set up is that you're using DNS to resolve to an internal VPC IP Address to access SES instead of the public one. so if you think it could be something to do with traversing the internet, this approach may resolve it.

answered 8 days ago
-2

Connection timeout issues with SES from your Elastic Beanstalk application could be related to several factors beyond DNS caching. Here are some areas to investigate:

Connection Timeout Configuration: The default connection timeout in the AWS SDK for Java 2.x is 2 seconds. This may be too low for your use case. Consider increasing the connection timeout - the recommended value is 1 second for in-region calls and 3 seconds for cross-region requests. Since you're calling SES in eu-west-1, ensure your timeout is appropriately configured.

JVM DNS TTL Settings: While you've disabled DNS caching at the application level, the JVM-wide setting should also be configured. The recommended TTL for AWS resources is between 5-60 seconds (not 0). Setting it to 0 might actually cause issues. You should set the networkaddress.cache.ttl property in your JVM's security configuration file ($JAVA_HOME/jre/lib/security/java.security for Java 8 or $JAVA_HOME/conf/security/java.security for Java 11+) to a value like 5 or 60 seconds rather than disabling it entirely.

Connection Pool Issues: Intermittent timeouts could indicate connection pool starvation or connection leaks. Check if you're properly managing your SesClient instances - they should be reused rather than created for each request, and ensure connections are being released back to the pool after use.

Network/Proxy Configuration: If you have any proxies configured in your environment, try bypassing them to see if that resolves the issue. Misconfigured proxies are a common cause of connection timeouts.

TCP Dump Analysis: If the issue persists, run a TCP dump at the client edge of your network to investigate potential network-level issues that might be causing the intermittent failures.
Sources
Troubleshooting FAQs - AWS SDK for Java 2.x
Set the JVM TTL for DNS name lookups - AWS SDK for Java 2.x
Setting the JVM TTL for DNS name lookups - Amazon Timestream

answered 8 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.