How do I troubleshoot instance metadata issues on my EC2 Linux instance?

Short description

Amazon EC2 locally accesses instance metadata within the instance through HTTP requests to the 169.254.169.254 IPv4 endpoint or [fd00:ec2::254] IPv6 endpoint. To access the instance metadata, you must use the Instance Metadata Service (IMDS). IMDSv1 doesn't require an authentication token, but IMDSv2 requires a session token for enhanced security.

When you retrieve instance metadata, you might encounter the following issues:

• HTTP request errors such as timeouts and HTTP 400 or 404 errors
• IMDSv2 token request failures
• Software-specific handling issues
• Misconfigured network interface or route table
• Proxy or NAT gateway configurations that block metadata access
• (IPv6 only) An IMDSv6 endpoint that isn't activated
• Missing or stale AWS Identity and Access Management (IAM) instance profile attachments
• Local firewalls such as iptables or firewalld that block access to 169.254.169.254
• High request volume that causes throttled metadata requests

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Resolve HTTP request errors

Take the following troubleshooting actions based on the HTTP error code that you receive.

(IMDSv1 only) "404 - Not Found" error

The HTTP 404 error occurs when you enter a URL that's not valid or when you updated your instance's IAM role but didn't refresh the role.

To resolve this error, verify that you're using the correct URL. Also, detach and then reattach your instance's IAM role.

Then, start and stop your instance to apply the changes.

(IMDSv2 only) "400 - Bad Request" error

The HTTP 400 error occurs when your PUT request uses a token that's not valid or when the software sends incorrect headers. For example, some FortiGate or Matillion agents don't cache or reuse IMDSv2 tokens.

To resolve this error, run the following command to generate a new token for your PUT request:

$ TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl

Also, check the system or application logs for long-running processes that need to refresh their tokens.

(IMDSv2 only) "401 - Unauthorized" error

The HTTP 401 error occurs when your GET request uses a token that's not valid.

To resolve this error, run the following command to generate a new token for your GET request:

$ curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/

Also, check the system or application logs for long-running processes that need to refresh their tokens.

"403 - Forbidden" error

The HTTP 403 error occurs when you deactivate IMDS at the instance level or occurs when a security group, firewall, or route table blocks instance access. The error can also occur if you must use IMDSv2, but the client uses IMDSv1.

To resolve this issue, run the following describe-instances AWS CLI command to check the IMDS configuration:

aws ec2 describe-instances --instance-ids your_instance_id --query 'Reservations[].Instances[].MetadataOptions'

Note: Replace your_instance_id with your instance ID.

If HttpEndpoint is set to disabled, then run the following modify-instance-metadata-options command to activate IMDS:

aws ec2 modify-instance-metadata-options --instance-id your_instance_id --http-endpoint enabled

Note: Replace your_instance_id with your instance ID.

Make sure that your configuration allows outbound HTTP access to 169.254.169.254 (for IPv4) or [fd00:ec2::254] (for IPv6) from the instance.

If your instance uses a proxy, NAT configuration, load balancer, or multiple internal network hops, then it's a best practice to configure HttpPutResponseHopLimit to 2 or higher. Configure a hop value that's high enough to allow the token response to make it through the network layers. By default, HttpPutResponseHopLimit allows 1 hop only. To increase this value, run the following modify-instance-metadata-options command:

aws ec2 modify-instance-metadata-options --instance-id your_instance_id --http-put-response-hop-limit 2

Note: Replace your_instance_id with your instance ID.

Check for proxy configuration issues

If your instance uses a proxy to access the internet, then you must exclude the IMDS IP address from proxy traffic. If you don't, then you might receive HTTP 403 and 404 errors or connection timeouts.

To exclude the IMDS IP addresses from the proxy, run the following command to set the no_proxy environment variable:

export no_proxy='169.254.169.254,[fd00:ec2::254]'

Note: Some applications, such as Matillion, Fortigate, or custom services, might not use the system-level no_proxy settings. In this scenario, configure no_proxy in the application. For a dual-stack configuration, make sure to exclude both the IPv4 and IPv6 metadata endpoints.

Make sure that you activated IPv6 support

If you use an IPv6-only subnet, then run the following modify-instance-metadata-options command to explicitly activate IPv6 support for IMDS:

aws ec2 modify-instance-metadata-options \
  --instance-id your_instance_id \
  --http-protocol-ipv6 enabled

Note: Replace your_instance_id with your instance ID.

If you use virtual private cloud (VPC) endpoints with strict security group rules, then make sure that they allow port 80 (HTTP) access to the metadata IP address.

For Fortigate or Matillion applications, confirm that the software supports IMDSv2 session tokens.

Troubleshoot stale network configuration, stale IAM role associations, or internal software issues

Reboot your instance.

Check your local firewall rules

To check whether local firewall blocks use iptables to block access to the IMDS endpoint, run the following command:

sudo iptables -L

Example output of a blocked endpoint:

Chain OUTPUT (policy ACCEPT)
target     prot opt source    destination
REJECT     tcp  --  anywhere  169.254.169.254  owner UID match 1000-10000 reject-with icmp-port-unreachable

To check for rules that block traffic to 169.254.169.254, run the following command:

curl http://169.254.169.254/latest/meta-data/

If a rule blocks traffic, then you receive an output similar to the following example:

curl: (7) Failed to connect to 169.254.169.254 port 80 after 0 ms: Connection refused

To remove the blocking rule, run the following command:

sudo iptables -D OUTPUT -p tcp -d 169.254.169.254 -m owner --uid-owner 1000-10000 -j REJECT

If your instance is dual-stack, then make sure that no IPv6 firewall rules block the [fd00:ec2::254] IPv4 address. If your iptables are empty, but traffic is still blocked, then check operating system (OS) firewall daemons such as firewalld or ufw. Also, check security agents, such as antivirus or firewall software, that might enforce hidden rules.

Check whether Amazon EC2 throttled your request

Amazon EC2 throttles traffic to the IMDS based on the number of packets per second (PPS). Each elastic network interface attached to the instance has a maximum quota of 1024 PPS for metadata-related traffic. If your PPS rate exceeds this quota, then you receive "HTTP 5xx" errors, the metadata retrieval fails, or the application times out.

To mitigate throttling issues, take the following actions:

• Implement exponential backoff and retry logic in your application when you access IMDS.
• Check with your vendor or update your software to the latest version to make sure that it supports IMDSv2.
• When you use IMDSv2, generate a token once and reuse it for multiple metadata queries during its Time-to-Live (TTL).
• Upgrade to the latest IMDSv2 version to make sure your configuration correctly implements token reuse and exponential backoff.
• Don't frequently poll instance metadata.
• Use instance caching in your application wherever possible.

If your software floods metadata requests in a tight loop without delay, then you might experience throttling or metadata failures. Use tcpdump, strace, or app debug logs to check for frequent repeat calls to 169.254.169.254. To monitor throttling events, run the following command to check the network interface driver for the linklocal_allowance_exceeded metric:

ethtool -S eth0

Note: Replace eth0 with your network interface name. In the output, check linklocal_allowance_exceeded for a value that isn't 0 to identify throttling.

Example output:

linklocal_allowance_exceeded: 245

The preceding example output shows that Amazon EC2 throttled 245 packets to IMDS because of an exceeded PPS quota.

Related information

Using a proxy on Amazon EC2 instances

Access instance metadata for an EC2 instance

Limit access to the Instance Metadata Service

Query throttling