How can I troubleshoot DNS resolution issues with my Route 53 private hosted zone?

Resolution

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you're using the most recent AWS CLI version.

Confirm that you activated DNS support in the VPC

To allow private hosted zone record resolution, you must turn on DNS support in your virtual private cloud (VPC). Verify that DNSSupport and DNSHostnames are set to True in your VPC.

Confirm that you associated the correct VPC ID with the private hosted zone

When you associate a private hosted zone with a VPC, Route 53 Resolver creates an auto-defined rule and associates it with the VPC. Resources in that VPC can query the Resolver to resolve DNS records in the private hosted zone.

Confirm that you associated the correct VPC ID with your private hosted zone. Also, be sure that you're querying the resource records of the domain from within the same VPC.

To get a list of VPCs associated with a hosted zone, run the following command in the AWS CLI:

aws route53  list-hosted-zones-by-vpc --vpc-id VPC_ID --vpc-region REGION_ID

Note: Replace VPC_ID and REGION_ID with your relevant values.

To get a list of private hosted zones that are associated with specific VPCs, run the following command in the AWS CLI:

aws route53 get-hosted-zone --id VPC_ID

Note: Replace VPC_ID with your relevant value.

Confirm that you configured forwarding rules for private hosted zone domains in custom DNS servers towards the Amazon-provided DNS server (CIDR+2).

If you configured custom DNS servers or Active Directory servers in the DHCP options for DNS in your VPC, then check the following configurations:

• In the forwarding rule, confirm that your servers forward private domain DNS queries to the IP address of the Amazon-provided DNS servers of your VPC. For example, if the primary CIDR range for your VPC is 172.31.0.0/16, then the VPC DNS server's IP address is 172.31.0.2. This is the VPC network range plus two.
• Confirm that the domain that you configured in the custom servers is different from your private hosted zone. If the domain is the same as your private hosted zone, then the server is authoritative for that domain. In this case, the server doesn't contact the Amazon-provided DNS server for private hosted zone domains.

Review custom settings in resolv.conf

If you experience intermittent DNS resolution or responses, then review the configuration settings of your source instance in resolv.conf .

For example, you configure the rotate option in resolv.conf to load balance DNS queries between the Amazon DNS server and the public Google resolver server (8.8.8.8). In this case, these are the resolv.conf settings:

options rotate; generated by /usr/sbin/dhclient-script
nameserver 8.8.8.8
nameserver 172.31.0.2

In your first query to the public Google resolver (8.8.8.8), you receive the expected NXdomain response. You receive this response because the resolver is trying to find the response in the public hosted zone instead of the private hosted zone:

Private hosted Zone Record - resolvconf.local\[ec2-user@ip-172-31-253-89 etc\]$ curl -vks http://resolvconf.local
\* Rebuilt URL to: http://resolvconf.local/
\* Could not resolve host: resolvconf.local

15:24:58.553320 IP ip-172-31-253-89.ap-southeast-2.compute.internal.40043 > dns.google.domain: 65053+ A? resolvconf.local. (34)
15:24:58.554814 IP dns.google.domain > ip-172-31-253-89.ap-southeast-2.compute.internal.40043: 65053 NXDomain 0/1/0 (109)

However, the second query resolves successfully. The second query is successful because it reaches the VPC DNS resolver that's associated to your private hosted zone:

[ec2-user@ip-172-31-253-89 etc]$ curl -vks http://resolvconf.local* Rebuilt URL to: http://resolvconf.local/
*   Trying 1.1.1.1...
* TCP_NODELAY set
* Connected to resolvconf.local (1.1.1.1) port 80 (#0)

15:25:00.224761 IP ip-172-31-253-89.ap-southeast-2.compute.internal.51578 > 172.31.0.2.domain: 7806+ A? resolvconf.local. (34)
15:25:00.226527 IP 172.31.0.2.domain > ip-172-31-253-89.ap-southeast-2.compute.internal.51578: 7806 1/0/0 A 1.1.1.1 (50)

Confirm that private hosted zones don't have overlapping namespaces

When multiple zones have overlapping namespaces (such as example.com and test.example.com), the Resolver routes traffic to the hosted zone based on the most specific match. If there's a matching zone but no record that matches the domain name and type in the request, then Resolver doesn't forward the request. Instead, Resolver returns NXDOMAIN (non-existent domain) to the client.

Confirm that you configured the correct record in the most specific private hosted zone for successful DNS resolution.

For example, let's say that you have two private hosted zones with the following records:

Private hosted zone name	Record name	Value
local	overlap.privatevpc.local	60.1.1.1
privatevpc.local	overlap.privatevpc.local	50.1.1.1

The request gets the following answer from the most specific matched private hosted zone:

[ec2-user@IAD-BAS-INSTANCE ~]$ dig overlap.privatevpc.local +short50.1.1.1

Confirm that there's no zone delegation configured in the private hosted zone

Private hosted zones don't support zone delegation. If delegation is configured, then the client gets the "Servfail" response code from the VPC resolver.

Use the AWS CLI to confirm that zone delegation isn't configured in the private hosted zone, such as in the following example:

Private hosted zone:abc.com Delegation NS record for: kc.abc.com Resource record: test.kc.abc.com

[ec2-user@ip-172-31-0-8 ~]$ dig test.kc.abc.com;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 63414
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;test.kc.abc.com        IN      A
;; Query time: 15 msec
;; SERVER: 172.31.0.2#53(172.31.0.2)
;; WHEN: Fri Apr 16 15:57:37 2021
;; MSG SIZE  rcvd: 48

Confirm that the resource record's routing policy is supported in private hosted zones.

Confirm that you configured a routing policy in your resource record that's supported by a private hosted zone:

• Simple routing
• Failover routing
• Multivalue answer routing
• Weighted routing
• Latency-based routing
• Geolocation routing

Confirm that the Resolver rule and its inbound endpoint resolve to different VPCs

When the outbound endpoint in a Resolver rule points to an inbound endpoint that shares a VPC with the rule, the result is a loop. In this loop, the query continually passes between the inbound and outbound endpoints.

You can still associate the forwarding rule with other VPCs that are shared with other accounts. To do this, use AWS Resource Access Manager (AWS RAM). Private hosted zones that are associated with the hub or a central VPC resolve from queries to inbound endpoints. A forwarding resolver rule doesn't change this resolution, such as in the following example:

Hub VPC: VPC A - CIDR 172.31.0.0/16 Spoke VPC: VPC B - CIDR 172.32.0.0/16 Inbound IP address: 172.31.253.100 and 172.31.2.100 Target IP addresses in forwarding rule: 172.31.253.100 and 172.31.2.100 Rule associated with VPCs: VPC A and VPC B Client: 172-32-254-37

ubuntu@ip-172-32-254-37:~$ dig overlap.privatevpc.local;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 9007
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; QUESTION SECTION:
;overlap.privatevpc.local. IN A
;; Query time: 2941 msec
;; SERVER: 172.32.0.2#53(172.32.0.2)

In this output, the DNS request continuously hops between the outbound and inbound endpoints. The request checks the rule that's associated with VPC A and sends the query back to the outbound endpoint. After several attempts, the query times out and responds with a Servfail response code.

To fix this issue and break the loop, remove the hub VPC association (VPC A) with the rule. Then, you get a successful response from the private hosted zone:

ubuntu@ip-172-32-254-37:~$ dig overlap.privatevpc.local;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58606
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;overlap.privatevpc.local. IN A
;; ANSWER SECTION:
overlap.privatevpc.local. 0 IN A 50.1.1.1
;; Query time: 5 msec
;; SERVER: 172.32.0.2#53(172.32.0.2)

Confirm that the on-premises resolver sends a recursive request

If you use queries from on-premises to Route 53 Resolver, then you can forward DNS queries from resolvers on your network to a VPC resolver. To do this, use the Resolver inbound endpoint. This action lets you resolve domain names for AWS resources, such as records in a private hosted zone.

In some cases, you might find that the private hosted zone isn't resolving successfully from the on-premises resolver. This behavior occurs because the on-premises resolver is sending an iterative query instead of a recursive request. The inbound endpoint is supporting recursive queries for successful DNS resolutions.

To verify the resolution type, use a packet capture on the DNS resolver (on-premises). Then, review the DNS flags (recursion desired = 0). To test the resolution, send an iterative request with the +norecurse dig command, or set norecurse with nslookup:

Inbound endpoint IP address: 172.31.253.150

On-premises Resolver IP address: 10.0.4.210

A failed iterative query to the inbound IP address results in the following output:

[ec2-user@IAD-BAS-INSTANCE ~]$ dig @172.31.253.150 overlap.privatevpc.local +norecurse; <<>> DiG 9.11.0rc1 <<>> @172.31.253.150 overlap.privatevpc.local +norecurse
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

A successful recursive query results in the following output:

[ec2-user@IAD-BAS-INSTANCE ~]$ dig @172.31.253.150 overlap.privatevpc.local;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19051
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;overlap.privatevpc.local.      IN      A
;; ANSWER SECTION:
overlap.privatevpc.local. 0     IN      A       50.1.1.1
;; Query time: 200 msec
;; SERVER: 172.31.253.150#53(172.31.253.150)

Confirm that you configured the correct rule priorities for the Amazon-provided DNS

When the client instance sends a query to the resolver (AWS-provided DNS server), the resolver verifies the instance's rules for where to route the request.

In general, the most specific rule takes priority. If there's a test.example.com Resolver rule and a longest.test.example.com private hosted zone, then lookup for the longest.test.example.com domain matches the private hosted zone.

If the rules are at the same domain level, then they have the following priority:

1. Resolver rule
2. Private hosted zone rule
3. Internal rule

For example, if there's a test.example.com Resolver rule and a test.example.com private hosted zone, then the Resolver rule takes priority. The query forwards to the servers or target IP addresses that are configured in the rule.

Related information

Working with private hosted zones

Which Amazon VPC options do I need to turn on to use my private hosted zone?

Avoid loop configurations with Resolver endpoints