How do I avoid DNS resolution failures in my Amazon EC2 Linux instance?

Short description

To avoid DNS resolution failures, apply a DNS cache.

When you use a DNS cache to query external DNS resources, the cache locally answers most of the recurring DNS queries. In this scenario, the cache doesn't interact with the DNS resolver over the network. You can query external DNS resources such as the following examples:

• Amazon Relational Database Service (Amazon RDS)
• Amazon ElastiCache
• Amazon Simple Storage Service (Amazon S3)

You can use the following resolution steps for all versions of Amazon Linux. If you use another distribution, then see the documentation for your distribution:

• For Debian, see Local caching on the Debian website.
• For Ubuntu, see Dnsmasq on the Ubuntu website.
• For Red Hat Enterprise Linux (RHEL), see How to configure DNS caching server with dnsmasq in RHEL on the Red Hat website.

Resolution

Note: The following resolution steps use 169.254.169.253 as the DNS resolver IP address. If you use a different DNS resolver, then replace 169.254.169.253 with your DNS resolver IP address.

Set up a local DNS cache with dnsmasq

To set up a local DNS cache, use dnsmasq. For more information, see dnsmasq on the thekelleys.org.uk website.

Complete the following steps:

1. Run the following command to install the dnsmasq server:

   sudo yum install -y dnsmasq

2. If your instance runs on Amazon Linux 2023 (AL2023), then proceed to step 3. To create a dedicated system user to run dnsmasq, run the following command:

   sudo groupadd -r dnsmasq
   sudo useradd -r -g dnsmasq dnsmasq

   Note: dnsmasq typically runs as the root user. However, the user changes after startup, and drops root permissions. By default, the user is nobody.

3. Run the following command to create a backup of the dnsmasq.conf file:

   sudo mv /etc/dnsmasq.conf /etc/dnsmasq.conf.ori

4. To create and open the /etc/dnsmasq.conf configuration file with the vim text editor, run the following command:

   sudo vim /etc/dnsmasq.conf

   Note: You can use any text editor, such as vi or nano, to modify the /etc/dnsmasq.conf file.

5. In the /etc/dnsmasq.conf file, enter the following content:

   # Server Configuration
   listen-address=127.0.0.1
   port=53
   bind-interfaces
   user=dnsmasq
   group=dnsmasq
   pid-file=/var/run/dnsmasq.pid
   
   # Name resolution options
   resolv-file=/etc/resolv.dnsmasq
   cache-size=500
   neg-ttl=60
   domain-needed
   bogus-priv

   Note: The bogus-priv option in dnsmasq causes a reverse DNS lookup to fail for private IP address ranges that aren't in /etc/hosts. This option can also cause a reverse DNS lookup to fail for the Dynamic Host Configuration Protocol (DHCP) leases file. To perform a successful reverse lookup, comment out or remove bogus-priv.

6. Create the /etc/resolv.dnsmasq file and set the Amazon DNS server or the custom domain name servers that you specified in the DHCP options sets. Run the following command:

   sudo bash -c "echo 'nameserver 169.254.169.253' > /etc/resolv.dnsmasq"

   Note: For more information about DNS server locations, see What is DHCP? You can also adjust the 169.254.169.253 server location when you create an Amazon Machine Image (AMI) from an instance with the dnsmasq cache to launch in another virtual private cloud (VPC) with a different CIDR.

7. To restart the dnsmasq server and set the service to start up on boot, run one of the following commands based on your distribution.
   Amazon Linux 1 (AL1):

   sudo service dnsmasq restart
   sudo chkconfig dnsmasq on

   Amazon Linux 2 (AL2) and AL2023:

   sudo systemctl restart dnsmasq.service
   sudo systemctl enable dnsmasq.service

8. To verify that dnsmasq works correctly, run the following command:

   dig aws.amazon.com @127.0.0.1

   If the response is similar to the following example, then the dnsmasq cache works as expected:

   ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.13.11 <<>> aws.amazon.com @127.0.0.1
   ;; global options: +cmd
   ;; Got answer:
   ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33958
   ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
   
   ;; OPT PSEUDOSECTION:
   ; EDNS: version: 0, flags:; udp: 1232
   ;; QUESTION SECTION:
   ;aws.amazon.com.                        IN      A
   
   ;; ANSWER SECTION:
   aws.amazon.com.         300     IN      CNAME   tp.8e49140c2-frontier.amazon.com.
   tp.8e49140c2-frontier.amazon.com. 9 IN  CNAME   dr49lng3n1n2s.cloudfront.net.
   dr49lng3n1n2s.cloudfront.net. 59 IN     A       108.158.61.67
   dr49lng3n1n2s.cloudfront.net. 59 IN     A       108.158.61.79
   dr49lng3n1n2s.cloudfront.net. 59 IN     A       108.158.61.96
   dr49lng3n1n2s.cloudfront.net. 59 IN     A       108.158.61.102
   
   ;; Query time: 2 msec
   ;; SERVER: 127.0.0.1#53(127.0.0.1)
   ...

9. Set the dnsmasq DNS cache as the default DNS resolver.
   Note: You must suppress the default DNS resolver that DHCP provides. To do this, change or create the /etc/dhcp/dhclient.conf file.
   For AL2, run the following command:

   sudo bash -c "echo 'supersede domain-name-servers 127.0.0.1, 169.254.169.253;' >> /etc/dhcp/dhclient.conf"

   For AL2023, run the following command:

   sudo bash -c 'echo "DNS=127.0.0.1" >> /etc/systemd/resolved.conf'
   sudo bash -c 'echo "DNS=169.254.169.253" >> /etc/systemd/resolved.conf'

   Note: AL2023 uses systemd-networkd as the default DNS resolver.

10. To apply the change, run the following command based on your Linux distribution:
    AL2:

    sudo dhclient

    AL2023:

    sudo systemctl restart systemd-resolved.service

    -or-
    To restart the network service, run the following command:

    sudo systemctl restart network

    -or-
    To reboot your instance, run the following command:

    sudo reboot

11. Run the following command to verify that your instance uses the DNS cache:

    dig aws.amazon.com

    If the output shows that the replying server is 127.0.0.1, then the DNS cache works as expected.
    Example output:

    ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.13.11 <<>> aws.amazon.com
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29129
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 1232
    ;; QUESTION SECTION:
    ;aws.amazon.com.                        IN      A
    
    ;; ANSWER SECTION:
    aws.amazon.com.         297     IN      CNAME   tp.8e49140c2-frontier.amazon.com.
    tp.8e49140c2-frontier.amazon.com. 25 IN CNAME   dr49lng3n1n2s.cloudfront.net.
    dr49lng3n1n2s.cloudfront.net. 41 IN     A       108.158.61.102
    dr49lng3n1n2s.cloudfront.net. 41 IN     A       108.158.61.96
    dr49lng3n1n2s.cloudfront.net. 41 IN     A       108.158.61.79
    dr49lng3n1n2s.cloudfront.net. 41 IN     A       108.158.61.67
    
    ;; Query time: 0 msec
    ;; SERVER: 127.0.0.1#53(127.0.0.1)   <-------------
    ...

Automate dnsmasq for new instances

To automate the installation and configuration of dnsmasq as a DNS resolver on Amazon Linux, use one of the following options:

• Use the AutomateDnsmasq.sh bash script. To download the file, see AutomateDnsmasq.sh on the GitHub website.
• Use the AutomateDnsmasq.cloudinit directives. To download the file, see AutomateDnsmasq.cloudinit on the GitHub website.

To automate the dnsmasq installation on other Linux distributions, use one of the preceding files to make the necessary customization. For either file, if you use the 169.254.169.253 Amazon DNS server alternative address, then the files can run on VPC instances.

To run either file at launch, enter the file contents in User data. You can use AWS Systems Manager Run Commands to run the bash script. It's a best practice to use the directives for only first-boot initialization.

To run the Bash script as a standalone script, complete the following steps:

1. To download the script on your instance and make it executable, run the following command:

   wget https://raw.githubusercontent.com/awslabs/aws-support-tools/master/EC2/AutomateDnsmasq/AutomateDnsmasq.sh
   chmod +x AutomateDnsmasq.sh

2. To run the script, run the following command as a root user:

   sudo ./AutomateDnsmasq.sh

Related information

Amazon EC2 instance IP addressing