How can I troubleshoot reverse DNS issues with Route 53 rules and outbound endpoints?

Short description

You can use reverse DNS lookup to return the domain name of an IP address. The format of a reverse DNS record falls under the namespace "in-addr.arpa.", where the subdomain is represented by the octets of the IP address in reverse order. The record type is pointer (PTR).

For example, suppose you have a virtual private cloud (VPC) with the CIDR range 172.31.0.0/16. In the VPC, you have an instance with an IP address of 172.31.2.23. The reverse zone for this VPC is 31.172.in-addr.arpa. The reverse record of this instance's IP address is 23.2.31.172.in-addr.arpa.

You can check the DNS response by running dig or nslookup. These tools allow you to test directly to the IP address and perform the reverse lookup on your behalf. In the following output examples, notice that the query is sent directly to the IP address 172.31.2.23. Both dig and nslookup perform a resolution against the name 23.2.31.172.in-addr.arpa.

Using the -x parameter with dig, you can perform a reverse DNS resolution. When you use this parameter, dig automatically adds the name, class, and type arguments. Refer to the QUESTION SECTION to see that dig automatically queried the correct name (23.2.31.172.in-addr.arpa.), class (IN), and record type (PTR).

dig -x 172.31.2.23:

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.amzn2.0.2 <<>> -x 172.31.2.23
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58812
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; 
OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;23.2.31.172.in-addr.arpa.    IN    PTR

;; 
ANSWER SECTION:
23.2.31.172.in-addr.arpa. 60    IN    PTR    testresolution.com.

An nslookup on an IP address performs a reverse lookup:

nslookup 172.31.2.23
23.2.31.172.in-addr.arpa    name = testresolution.com.

Resolution

Understand the query flow

It's important to understand the query flow when using resolver rules on a VPC and outbound endpoints. Understanding the flow can help you identify where potential issues might occur and guide your troubleshooting approach. The flow is as follows:

1. A client in the VPC performs a reverse DNS resolution for an address within the same VPC.
2. The query arrives at the VPC DNS resolver (VPC CIDR range plus 2).
3. The query is matched by a resolver rule on the VPC.
4. The query is sent by the outbound endpoint to the target IP addresses over a VPN or AWS Direct Connect connection.
   Note: The target IP addresses are on-premises DNS servers that manage the reverse DNS for the CIDR range of the VPC.

Identify the expected and actual DNS responses

Use dig and nslookup to perform queries directly to the IP address of your on-premises DNS server. These tools try to resolve the correct record name. Be sure to note the actual DNS response code that's returned. Keep in mind that an unexpected response code might not indicate an issue with the rule or endpoint configuration. For example:

• NXDOMAIN might be an unexpected but valid DNS response. This response indicates that the server that's being queried doesn't contain the requested record.
• SERVFAIL indicates that there's a timeout or other issue on the path of the query. This response requires further investigation, as described later in this article.
• An unexpected answer in the ANSWER SECTION might indicate that a different rule was used.

Determine if the query arrives at the VPC DNS resolver

For a query to be matched by a rule on a VPC, the query must arrive at the VPC DNS resolver. Check the VPC settings to confirm that DNS Support is enabled.

Refer to the server fields in both dig and nslookup to check the resolver IP address.

dig:

;; SERVER: 172.31.0.2#53(172.31.0.2)

nslookup:

Server:		172.31.0.2

Find the most specific rule that's being matched

When the query arrives at the VPC DNS resolver, it must be matched by a rule on that VPC. When rules are evaluated, the most specific rule is matched. Internally, rules are created to cover all reverse lookups of every IP address in the source and connected VPCs. These rules don't display in the AWS Management Console. These rules are known as autodefined rules and they're created by Route 53 Resolver.

1. Identify any autodefined rules on the VPC and connected VPCs. If you have peered VPCs or VPCs connected through a transit gateway (and DNS support is enabled), make note of all rules created for the reverse resolution of each connected CIDR.
   Note: Resolver creates these autodefined rules when DNS Hostnames is set to true. If you want to override an autodefined rule, you can create a conditional forwarding rule for the same domain name.
2. If DNSSupport and DNSHostnames are enabled, note any private hosted zones associated to the VPC.
   Note: If the resolver forwarder rule and private hosted zone are overlapping, the resolver rule takes precedence. Then, the query is forwarded to the on-premises server.
3. Compare your list of rules and associated private hosted zones against the query to determine which rule is selected and where the query goes.

Troubleshoot your outbound endpoints

• Confirm that your outbound endpoints are configured to send the query to the target IP addresses specified in the rule.
• Confirm that the security group used by the outbound endpoints allow outbound TCP and UDP traffic to the IP addresses and ports of the on-premises DNS server.
• Confirm that the access control lists (ACLs) allow TCP and UDP traffic to the IP addresses and ports of the on-premise DNS server. The ACLs must also allow traffic to the ephemeral ports (1024-65535).
• Confirm that the route tables for the outbound endpoints have a route for the IP addresses of the on-premises server to the VPN or Direct Connect connection.

Check if the outbound endpoint can send the query over the connection specified in its route table

You can run a dig or nslookup directly to the IP address of the on-premises DNS resolver to verify that the VPN or Direct Connect connection is allowing communication. You can also send a ping to an on-premises host that allows internet control message protocol (ICMP) to rule out issues on the connection.