How do I determine whether my DNS queries to the Amazon DNS server fail because of VPC DNS throttling?

Short description

The Amazon DNS quota is 1,024 packets per second (PPS) for each elastic network interface. If you exceed the quota, then Amazon Route 53 rejects the traffic.

VPC Flow Logs doesn't capture the traffic that applications send to Amazon DNS servers. To identify the cause of DNS query failures, use packet captures or Traffic Mirroring.

Note: Route 53 query logs capture only the traffic that reaches the VPC.2 resolver AmazonProvidedDNS. Throttled DNS queries don't appear in query logs because the queries throttle at the network interface level.

Resolution

Determine the source of the DNS query failures

Use tcpdump (Linux only)

To use tcpdump, complete the following steps:

1. Run the following command on your Amazon Elastic Compute Cloud (Amazon EC2) instance: 

   TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
   INSTANCE_ID=`curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id`
   sudo tcpdump -i eth0 -s 350 -C 100 -W 20 -w /var/tmp/$INSTANCE_ID.$(date +%Y-%m-%d:%H:%M:%S).pcap

   Note: The preceding command captures the initial 350 bytes of the packet, saves 20 files of 100 MB each, and overwrites the old packet captures. If your primary network interface isn't eth0, then replace eth0 in the preceding command with your primary network interface. To find your primary interface, run the following command:

   netstat -i

2. Run the following command to count the number of sent DNS queries:

   tcpdump  -r <file_name.pcap> -nn dst port 53 | awk -F " " '{ print $1 }' | cut -d"." -f1 | uniq -c

Use Traffic Mirroring

If you can't use tcpdump, then use Traffic Mirroring to identify whether DNS queries are throttling.

Note: Traffic Mirroring is available for Nitro-based instances and certain non-Nitro instance types. Traffic Mirroring charges apply.

To capture and analyze traffic data, complete the following steps:

1. Complete the Traffic Mirroring prerequisites.
2. Create a traffic mirror target. Confirm that the target network interface or Network Load Balancer allows inbound traffic on port UDP 4789.
3. Create a traffic mirror filter. For Filter settings, confirm that amazon-dns is turned on for Network services - optional.
4. Create a traffic mirror session.

Traffic Mirroring gathers mirrored traffic and stores it on the traffic mirror target.

Note: Traffic Mirroring is a live stream of data. To capture the mirrored packets on the target and save them in a .pcap file, capture traffic on UDP port 4789.

Use Wireshark

Complete the following steps:

1. Open the captured traffic in Wireshark
   Note: To download Wireshark, see the Download page on the Wireshark website.
2. Choose the Statistics tab.
3. Select I/O Graph and clear all options.
4. Under Display Filter, use the VXLAN Network Identifier and the DNS query to add a filter. The QR field in the DNS header is 0. For example, if the VXLAN Network Identifier is 16777215, then the display filter for the graph is (vxlan.vni == 16777215) && (dns.flags.response == 0).
5. Review the graph. If it remains at about 1024 PPS, then throttling is occurring on the mirrored source.

Check the ENA driver network performance metric

If the EC2 instance is running one of the following Elastic Network Adapter (ENA) driver versions, then check the real-time metrics for DNS throttling:

• Linux: 2.2.10 or later
• Window: 2.2.2.0 or later

Check the linklocal_allowance_exceeded metric to determine the number of dropped packets because traffic exceeded the PPS quota for the following local services:

• Amazon VPC DNS Service
• Instance Metadata Service (IMDS)
• Amazon Time Sync Service

Check the metric at multiple intervals to determine whether the count is increasing. The linklocal_allowance_exceeded metric is cumulative from the last driver restart because of a stop and start or reboot. There's an issue only when the metric is increasing.

To get the linklocal_allowance_exceeded value, run the following command for eth0:

ethtool -S eth0 | grep link

For instances with the interface named ens5:

ethtool -S ens5 | grep link

Resolve DNS throttling issues

If DNS throttling is causing the failures, then take one of the following actions to resolve the issue:

• Activate DNS caching on the instance.
• Increase the DNS retry timer on the application.

Related information

Understand traffic mirror target concepts

Work with Traffic Mirroring to copy network traffic