How do I troubleshoot DNS resolution issues with Route 53 VPC Resolver endpoints?

Resolution

Troubleshoot Route53 inbound endpoints

To allow DNS resolvers on your network to use your inbound endpoint to forward DNS queries to Route 53 VPC Resolver, take the following actions:

• If your on-premises DNS server must forward DNS queries to the inbound endpoint for your domains, then create a conditional forwarding rule. You must create the rule on your on-premises DNS server. This configuration applies to private hosted zones and public domains.

• Confirm that you have connectivity to the inbound resolver endpoint IP addresses over the AWS Direct Connect connection or VPN. This step validates whether you can reach the inbound resolver endpoint IP address from your on-premises network. Use the following telnet command to test connectivity between the inbound endpoint resolver IP address on port 53:

  telnet yourInboundEndpointResolverIPAddress 53.

  Note: Replace yourInboundEndpointResolverIPAddress with your information.

• Check the security group associated with the inbound resolver endpoint. The security group must allow traffic on TCP and UDP port 53 from your on-premises DNS server IP address.

• Confirm that custom network access control lists (network ACLs) for the subnet that contains the inbound endpoint allow the following:
  Inbound UDP and TCP traffic from the on-premises DNS server on port 53.
  Outbound UDP and TCP traffic to the on-premises DNS server on destination port range 1024-65535.

• Confirm that the route table associated with the subnet where you created the inbound endpoint resolver includes a route to the on-premises network. Configure the route through a Direct Connect connection or VPN. This route allows the inbound endpoint resolver to return a DNS query response.

• To validate the domain resolution, complete a domain name lookup from the on-premises DNS server or localhost.

• Note: Replace yourPrivateHostedZoneDomainName and yourInboundEndpointIPAddress with your information.
  For Windows, run the following command:

  nslookup yourPrivateHostedZoneDomainName

  For Linux or macOS, run the following command:

  dig yourPrivateHostedZoneDomainName 

• If the preceding commands fail to return a record, then send the DNS query directly to the inbound resolver endpoint IP address. This bypasses your on-premises DNS server. 
  For Windows, run the following command:

  nslookup yourPrivateHostedZoneDomainName yourInboundEndpointIPAddress

  For Linux or macOS, run the following command:

  dig yourPrivateHostedZoneDomainName @ yourInboundEndpointIPAddress 

• Confirm that your on-premises DNS server sends only recursive queries. Route 53 inbound resolver doesn't support iterative queries.

• If you're resolving in a private hosted zone, then confirm that you associate your inbound resolver endpoints and private hosted zone with the correct Amazon Virtual Private Cloud (Amazon VPC).

Troubleshoot Route53 outbound endpoints

To make sure that Route 53 VPC Resolver uses your outbound endpoint to conditionally forwards queries to resolvers on your network, complete the following steps.

Test the outbound endpoint

To test outbound forwarding, query the Amazon VPC DNS Resolver IP address. This address is the base of your Amazon VPC CIDR, plus 2.

For Linux or macOS, run the following command:

dig example.com @10.0.0.2

For Windows, run the following command:

nslookup example.com 10.0.0.2

Note: Replace 10.0.0.2 with the DNS Resolver IP address of your Amazon VPC.

Note: Don't query the outbound endpoint ENI IP address directly. Queries sent directly to the outbound ENI IP address silently time out.

If the IP address returns, then the forwarding rule matched and the target DNS server responded successfully. Otherwise, take the following actions:

• If the return value is SERVFAIL, then the outbound endpoint can't reach the target DNS server, or the target returned no response. Check your security groups, network ACLs, and routing. Also check whether a DNS Firewall rule is blocking the query.
• If the return value is NXDOMAIN, then the query reached the target DNS server, but the domain doesn't exist. Check the domain in the forwarding rule.
• If the query times out, then check that you associated the forwarding rule with the Amazon VPC and the outbound endpoint is operational. If your configuration meets these conditions and your query still times out, then contact Support to troubleshoot.

Identify custom DNS servers

If your Amazon VPC DHCP options set specifies a custom DNS server instead of AmazonProvidedDNS, then queries from Amazon Elastic Compute Cloud (Amazon EC2) go to that custom server first. Queries don't go to the Route 53 VPC Resolver. Forwarding rules don't run unless queries reach the (.2) address.

To identify the DNS server your instance uses, run the following command on Linux:

cat /etc/resolv.conf

If your instance uses a custom DNS in the same Amazon VPC, then configure a conditional forwarder on your custom DNS server. Use this forwarder to forward the relevant domains to the Amazon VPC DNS Resolver IP address (.2).

If your instance uses a custom DNS in a different Amazon VPC or on-premises, then the .2 resolver is reachable only from within its own Amazon VPC. Create a Route 53 Resolver inbound endpoint in the workload Amazon VPC. Configure your custom DNS server to forward to the inbound endpoint ENI IP addresses. Route these IP addresses across Amazon VPC peering, AWS Transit Gateway, Direct Connect, or VPN.

Check network connectivity to your on-premises DNS server

For an on-premises DNS server, check the security group associated with the outbound resolver endpoint. The egress rule must allow UDP and TCP port 53 traffic to the IP addresses of your server.

Review the custom network ACL rules on the subnets where you created the outbound endpoint interfaces. These rules must allow the following:

• Outbound UDP and TCP traffic to the on-premises DNS server on port 53.
• Inbound UDP and TCP traffic from the on-premises DNS server on the ephemeral port range 1024 - 65535.

Confirm that the route table for the outbound endpoint subnet has a route to your on-premises DNS server through a Direct Connect connection or VPN. Review your on-premises DNS server's firewall configuration. The firewall must allow traffic from the outbound resolver endpoint IP addresses.

Note: You don't require network connectivity between your Amazon VPCs to forward DNS queries to a Resolver rule. This applies to Amazon VPCs in the same or different accounts. You require network connectivity only between your outbound endpoint's Amazon VPC and the remote DNS resolvers.

Review Resolver rule configurations

Take the following actions to check for misconfigured Resolver rules:

• If you use a shared rule, then make sure that you associate the shared rule with your Amazon VPC.
• The outbound endpoint referenced in the shared rule belongs to the rule owner's account. Confirm that you meet the network connectivity requirements from the owner's outbound endpoint to the target DNS server. Network connectivity requirements include security groups, network ACLs, routing, and on-premises firewall rules.
• Check for a Resolver rule that routes traffic to your network for the same domain name. This domain name takes precedence over private hosted zones.
• Check for rules with domain names that are more specific than your intended domain. Resolver uses the rule that contains the most specific domain name to route outbound DNS queries.

For more information, see How VPC Resolver determines whether the domain name in a query matches any rules.

Troubleshoot intermittent failures under high query volume

Security group connection tracking can reduce outbound endpoint throughput from 10,000 queries per second (QPS) per ENI to as low as 1,500 QPS per ENI. Symptoms include intermittent SERVFAIL responses or timeouts that correlate with peak traffic. For more information, see Security group connection tracking.

To avoid connection tracking, use the following permissive security group rules on the outbound endpoint, as recommended in Resolver endpoint scaling best practices:

• Ingress: TCP and UDP, all ports, from 0.0.0.0/0.
• Egress: TCP and UDP, port 53, to 0.0.0.0/0.

Debug with VPC Flow Logs and Resolver query logging

Use VPC Flow Logs to capture network flow information on the outbound endpoint's elastic network interfaces. Confirm whether traffic is leaving the endpoint and whether responses are returning.

Use Route 53 Resolver query logging to see query forwards, matches of forwarding rules, and the response status.