How do I troubleshoot NTP synchronization issues on my Amazon EC2 Linux instance?

Short description

To troubleshoot NTP synchronization issues in your EC2 Linux instance, first check whether the issue is clock drift or a time zone and display mismatch. Then, identify the time synchronization service that your instance uses, validate the configured NTP sources, and check whether the instance can reach those sources. It's a best practice to use the local Amazon Time Sync Service.

Resolution

Note: Commands and configuration file paths might vary based on your distribution and image build.

Check whether the issue is clock drift or time zone interpretation

To check the instance's configured local time zone, run the following command:

date

To check the instance's UTC time, run the following command:

date -u

To check the configured time zone, universal time, real-time clock (RTC) time, and system clock synchronization status, run the following command:

timedatectl

To compare the hardware clock with the system clock, run the following command:

hwclock --show

Check the command outputs to identify your issue:

• If date -u matches actual UTC but date shows an unexpected time, then the issue is time zone configuration, not NTP synchronization.
• If the hardware clock differs from the system clock, then check the hardware clock configuration, RTC configuration, and synchronization behavior.
• If timedatectl has System clock synchronized: no or NTP service: inactive in the output, then proceed to the next section.

Identify and troubleshoot the active time synchronization service

Important: Run only one time synchronization service at a time to avoid conflicts and unpredictable synchronization behavior.

Identify the time synchronization service that you use

It's a best practice to use chrony instead of ntpd for time synchronization because chrony synchronizes the system clock faster and with better accuracy. For more information, see Differences between chrony and ntpd on the Red Hat website. For information about NTP server configuration, see Configuring NTP using the chrony suite or Configuring NTP using ntpd on the Red Hat website.

Note: Some Linux distributions might use systemd-timesynd by default.

To identify the time synchronization service that your instance uses, run the following commands:

sudo systemctl status chronyd --no-pager
sudo systemctl status ntpd --no-pager
sudo systemctl status systemd-timesyncd --no-pager
systemctl is-active chronyd ntpd systemd-timesyncd

In the output, check for the service with the active status.

Troubleshoot chrony

To check the synchronization state, offset, and leap status, run the following command:

chronyc tracking

To check the configured NTP sources and confirm that chrony selected a valid source, run the following command:

chronyc sources -v

To check for recent chrony errors or warnings, run the following command:

journalctl -u chronyd --since "24 hours ago" --no-pager

Troubleshoot ntpd

To check whether the service is synchronized, run the following command:

ntpstat

To check the configured peer and source selection, run the following command:

ntpq -p

To check for recent ntpd errors or warnings, run the following command:

journalctl -u ntpd --since "24 hours ago" --no-pager

Troubleshoot systemd-timesyncd

To check whether the system clock is synchronized and the NTP service is active, run the following command:

timedatectl status

To check the time server and synchronization details, run the following command:

timedatectl timesync-status

To check for recent service errors or warnings, run the following command:

journalctl -u systemd-timesyncd --since "24 hours ago" --no-pager

(Optional) Migrate to chrony

To remove ntpd and install chrony on your Amazon Linux 2023 (AL2023) instances, run the following commands:

sudo dnf erase ntp*
sudo dnf install chrony

Note: For Amazon Linux (2) instances, replace dnf with yum.

To start and activate chrony, run the following commands:

sudo systemctl start chronyd
sudo systemctl enable chronyd

Configure the NTP sources

Check the configuration file for the list of configured NTP servers. The default location for the configuration file is /etc/chrony.conf for chrony and /etc/ntp.conf for ntpd. However, Amazon Linux can also use source directories and generated source files.

Identify the source directory

For AL2023, chrony can read NTP source configuration from the main /etc/chrony.conf file, and from source directories such as /etc/chrony.d/ or /run/chrony.d/. Before you add or change an NTP source, identify the source directories that exist and whether /etc/chrony.conf references them.

To check whether /etc/chrony.conf includes a sourcedir directive and view the source directory path, run the following command:

grep -n "^sourcedir" /etc/chrony.conf

To check whether the source directories exist and list existing source files, run the following command:

ls -l /etc/chrony.d /run/chrony.d 2>/dev/null

To check distribution-specific guidance, run the following command:

cat /etc/chrony.d/README 2>/dev/null

Then, create your own chrony source file in one of the identified source directories, such as /etc/chrony.d/your-source-file.sources. Note where you configured sources and make sure that you use the correct servers or pools.

For AL2, chrony can use source files from the /etc/chrony.d/ directory and the main /etc/chrony.conf file. Before you add or change an NTP source, check /etc/chrony.conf and /etc/chrony.d/README to check where to configure chrony source files. For example, if your instance uses /etc/chrony.d/ for chrony sources, then create or update a source file in that directory, such as /etc/chrony.d/your-source-file.sources. Then, add the NTP source to that file and restart chronyd.

Sync the NTP to the Amazon Time Sync Service

Sync the NTP to either the local Amazon Time Sync Service or the public Amazon Time Sync Service. It's a best practice to use the local Amazon Time Sync Service on your instances. Use the public Amazon Time Sync Service as a backup, or for resources that are outside of the Amazon Virtual Private Cloud (Amazon VPC).

You can also sync the NTP to the NTP server that's internal to your organization or to a trusted internet-accessible NTP server. If you use a host name, then make sure that DNS resolution works. If you use an external NTP source, then make sure that the instance allows outbound UDP traffic through port 123.

To add your own NTP source, open the ntpd or chrony configuration file where the sources are located. Then, add the following line:

server internal-ntp-server-hostname-or-ip-address

Note: Replace internal-ntp-server-hostname-or-ip-address with the name of your NTP server host name or IP address.

Restart the service after you change sources

Run the following command to restart the service and verify that the instance is running:

sudo systemctl restart chronyd.service
sudo systemctl status chronyd.service --no-pager

Note: The preceding command updates the chrony service. For ntpd, replace chronyd.service with ntpd.service.

(Optional) Override the default chrony configuration

By default, chrony can use Amazon Time Sync Service sources through link-local and remote endpoints.

To override the default configuration for AL2023, modify the /etc/sysconfig/chronyd file to include USE_AMAZON_NTP_POOL="no". Then, run the following command to restart chrony:

sudo systemctl restart chronyd.service

For AL2, modify the /etc/chrony.d/link-local.sources file. Add a # at the beginning of the line to comment out the entry for Amazon Time Sync Service. Example line:

#server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4

Then, run the following command to restart the chrony:

sudo systemctl restart chronyd.service

Note: The /etc/chrony.d/README file provides information about other ways to override the default configuration.

Confirm that your instance is synchronized to the correct time

To confirm that your instance is synchronized to the correct time, run the following commands based on your source.

chrony:

chronyc tracking
chronyc sources -v
timedatectl

ntpd:

ntpstat
ntpq -p
timedatectl

systemd-timesyncd:

timedatectl status
timedatectl timesync-status

In the output, check for the following settings to confirm that your instance is synchronized:

• timedatectl shows System clock synchronized: yes.
• timedatectl shows NTP service: active.
• chronyc tracking shows Leap status: Normal.
• chronyc sources -v shows a selected source.
  Note: Look for sources that are marked with an asterisk (*).
• ntpstat shows that the system is synchronized to an NTP server:

Example output where there are still synchronization issues:

System clock synchronized: no
NTP service: inactive
server not synchronized
host name not found
No suitable source for synchronization

Note: A failed ICMP ping to 169.254.169.123 doesn't always mean that Amazon Time Sync Service is unavailable or that time synchronization failed. NTP uses UDP port 123, so an ICMP ping doesn't test NTP synchronization.

Troubleshoot the "server not synchronized" error

If you receive the "server not synchronized" error from ntpstat, then check your ntpd configuration.

Run the following command to check the ntpd service status, peer selection, and service logs:

ntpstat
sudo systemctl status ntpd --no-pager
ntpq -p
journalctl -u ntpd --since "1 hour ago" --no-pager

In the output, check for errors such as "host name not found", "Name or service not known", or "Temporary failure in name resolution". To troubleshoot these issues, confirm that NTP host name can resolve.

To check whether the operating system (OS) can resolve the configured NTP host name, run the following command:

getent hosts ntp-server-hostname

Note: Replace ntp-server-hostname with the host name that's configured in /etc/ntp.conf, such as time.aws.com.

To check the DNS resolution of the NTP host name, run one of the following commands:

nslookup hostname

-or-

dig 0.rhel.pool.ntp.org

Note: Replace hostname with the host name that's configured in /etc/ntp.conf.

To check whether the instance has valid name server entries, run the following command:

cat /etc/resolv.conf

If /etc/resolv.conf has no valid nameserver, then NTP pools based on host names, such as 0.rhel.pool.ntp.org or time.aws.com, might not resolve. To resolve this issue, configure the instance to use a valid DNS resolver. Confirm that the instance's Dynamic Host Configuration Protocol (DHCP) option set includes a valid domain name server, such as AmazonProvidedDNS or your organization's DNS resolver. To apply your changes, reboot the instance.

If you don't need to use host names for your NTP source, then use the 169.254.169.123 local Amazon Time Sync Service IP address instead.

Troubleshoot issues where chronyd is active but AL2023 still doesn't synchronize

On AL2023, chrony can read sources from multiple locations. If chronyd is active but the instance doesn't synchronize, then check for duplicate, stale, or conflicting source definitions.

Complete the following steps:

1. Run the following command to check the source directories that /etc/chrony.conf references:

   grep -n "^sourcedir" /etc/chrony.conf

2. To list the files in the source directories, run the following command:

   ls -l /etc/chrony.d /run/chrony.d 2>/dev/null

3. To search the main chrony configuration file and source directories for configured NTP sources, run the following command:

   grep -R "169.254.169.123\|server\|pool" /etc/chrony.conf /etc/chrony.d /run/chrony.d 2>/dev/null

   In the output, check for duplicate, stale, or conflicting entries across etc/chrony.conf, etc/chrony.d/, and /run/chrony.d/.

4. To check the NTP sources that chrony uses, run the following command:

   chronyc sources -v	

5. Update the configuration to add the correct source or remove duplicate, stale, or conflicting data.

6. Run the following command to restart chrony and make chrony select a valid source:

   server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4
   sudo systemctl restart chronyd
   chronyc tracking
   chronyc sources -v
   timedatectl

Troubleshoot issues where AWS CLI or SDK commands fail with signature or timestamp errors

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

API request signing is time sensitive. If the instance clock is too far ahead or behind, then AWS CLI or SDK requests might fail. This issue occurs even if your credentials and AWS Identity and Access Management (IAM) permissions are correct. As a result, you encounter the following issues:

• The requested timestamp is too skewed.
• Signature verification fails.
• There's a signature mismatch.
• The requested timestamp expires.

To resolve this issue, check your IAM policies, credentials, or SDK signing logic for configuration issues.

Troubleshoot external or organization-managed NTP server

If the instance uses a public or corporate NTP server, then run the following command to identify the configured NTP source:

grep -E "^(server|pool)" /etc/chrony.conf /etc/ntp.conf 2>/dev/null

Then, run the following command to test UDP connectivity to the NTP server:

nc -zvu ntp-server-hostname-or-ip-address 123

Note: Replace ntp-server-hostname-or-ip-address with the NTP hostname or IP address.

If the connection fails, then update your configuration to allow outbound UDP traffic to the NTP server on port 123. Check your DNS settings, route tables, OS firewall rules, security group rules, and network access control list (network ACL) rules. Also, make sure to allow inbound UDP traffic from the NTP server to the instance on ephemeral ports 1024-65535. For network ACLs and stateless firewalls, make sure to allow both inbound and outbound traffic.

Troubleshoot large offset or slow corrections

If the time source is correct but the offset is large, then chrony might slowly skew the clock.

To check the offset, run the following command:

chronyc tracking

It might take time for chrony to gradually correct a large skew. To immediately correct the issue, run the following command to force a one-time clock step:

sudo chronyc makestep
sudo systemctl restart chronyd

Important: Sudden time changes can affect databases, clustered applications, distributed systems, logs, scheduled jobs, and monitoring.

Troubleshoot Amazon EC2 Auto Scaling instances with time issues

If you launched your instances through an Amazon EC2 Auto Scaling group, then fix the source to resolve time synchronization issues. Don't fix only the instances.

Check for and fix issues in the following sources:

• Launch templates
• User data scripts
• Custom Amazon Machine Image (AMI) or golden image pipelines
• Cloud-init scripts
• State Manager, a capability of AWS Systems Manager, associations
• Ansible, Chef, Puppet, or other configuration management tool settings
• Bootstrap scripts that modify /etc/chrony.conf, /etc/chrony.d/, or /run/chrony.d/

Then, launch a test instance and run the following commands to confirm that the instance has the expected time synchronization configuration:

timedatectl
chronyc tracking
chronyc sources -v
grep -R "server\|pool\|169.254.169.123" /etc/chrony.conf /etc/chrony.d /run/chrony.d 2>/dev/null

Related information

Precision clock and time synchronization on your EC2 instance