Random Error (getaddrinfo ENOTFOUND) on HTTPS request to EC2

Question

So I set up a EC2 instance to accept incoming HTTPS requests (using Node JS). The EC2 is managed by a load balancer, and is using a SSL certificate that I set up with the Certificate Manager and Route 53. The EC2 and load balancer are both allowing traffic on ports 80 (HTTP) and 443 (HTTPS). In the load balancer, traffic to port 80 is redirected to port 443.

Here's the issue...

Every so often (about after 15-20 HTTPS requests to the EC2), I get this error:
**getaddrinfo ENOTFOUND**

This error, however, does not happen when I make HTTP requests directly to the EC2 and bypass the load balancer.

I've checked the load balancer logs, and when the error occurs, nothing is shown in the load balancer logs. The logs only show successful requests.
This tells me that the bad request is not even getting to the load balancer, and it is probably a DNS issue (DNS is not being resolved).

This **getaddrinfo ENOTFOUND** kind of happens randomly.
Why does this happen, and how do I fix this?

oreoaws · Answer

There are multiple reasons why you can experience intermittent DNS resolution issues. The architecture you have will determine what troubleshooting steps are needed. For example, if a custom DNS server (customer managed) being used, here are a couple of things to check.

1. Does the DNS server return the answers all the time? 
2. Is there a network connectivity issue between the EC2 instance and the custom DNS server?
Depending on the answers, you may have to troubleshoot further on the respective DNS server or the network connection.

On the other hand, let's assume that the Route 53 Resolver (AmazonProvidedDNS) is being used. Amazon-provided DNS servers enforce a limit of 1024 packets per second per elastic network interface. Amazon provided DNS servers reject any traffic exceeding this limit. If your EC2 instance is running specific ENA driver versions listed here [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-network-performance-ena.html], you can review the real-time metrics for DNS throttle using the linklocal_allowance_exceeded metric.

To retrieve the metric linklocal_allowance_exceeded value, run the following command:
ethtool -S eth0

If you confirm DNS throttling is the cause of the issue, you can resolve this by enabling DNS caching on the EC2 instance, so the cache can respond to recurring DNS queries locally and thus reduce the number of DNS queries being sent to the Route 53 Resolver.

Here are some reference articles with more information on how to troubleshoot and resolve DNS throttling 
https://repost.aws/knowledge-center/vpc-find-cause-of-failed-dns-queries
https://repost.aws/knowledge-center/dns-resolution-failures-ec2-linux

Random Error (getaddrinfo ENOTFOUND) on HTTPS request to EC2

Add your answer

Relevant content