How can I troubleshoot instance metadata issues on my EC2 Linux instance?

Short description

Instance metadata is information about the EC2 instance, such as instance ID, public and private IP addresses, security groups, AWS Identity and Access Management (IAM) roles, and so on. The instance metadata is exposed to the instance through HTTP requests to 169.254.169.254.

You might encounter the following issues when retrieving instance metadata from the Instance Metadata Service (IMDS):

• HTTP request errors
• Proxy configuration
• Local firewall rules
• Request throttling

Resolution

HTTP request errors

When making an HTTP request to retrieve instance metadata, the following HTTP error codes might be returned:

HTTP 404 - Not Found

The HTTP 404 error applies to IMDS v1. This error occurs when the requested resource isn't available. Verify that you're using the correct URL.

400 - Bad Request

The 400 - Bad Request error applies to IMDS v2. This error occurs if the PUT request is not valid.

401 - Unauthorized

The 401 - Unauthorized error occurs if the GET request uses a token that's not valid. If this occurs, generate a new token.

403 - Forbidden

The 403 - Forbidden error occurs if the request isn't allowed or IMDS is turned off. Run the following command to check IMDS status. In the following example command, replace your_instance_ID with the correct value for your use case.

$ aws ec2 describe-instances -instance-ids  -query 'Reservations[].Instances[].MetadataOptions'

The following is example output from the preceding command:

[
  {
    "State": "applied",
    "HttpTokens": "optional",
    "HttpPutResponseHopLimit": 1,
    "HttpEndpoint": "disabled",
    "HttpProtocolIpv6": "disabled",
    "InstanceMetadataTags": "disabled"
  }
]

If the command option shows that the HttpEndpoint is turned off, then run the following command:

aws ec2 modify-instance-metadata-options \
    --instance-id  \
    --http-endpoint enabled

Proxy configuration

If you're using a proxy to access the Internet, then you must exclude the IMDS IP address (169.254.169.254). If the IMDS IP address isn't excluded, then you might not be able to retrieve instance metadata.

To exclude the IMDS IP address from proxy use, set a NO_PROXY environment variable with the following address:

export NO_PROXY=169.254.169.254

Local firewall rules

Firewalls in the instance might prevent some or all processes from accessing the IMDS.

Make sure that firewalls in the instance's operating system aren't blocking outgoing traffic to the instance metadata IP address. Firewalls include iptables, UFW (uncomplicated firewall), and so on.

Use the following command to check firewall rules for iptables

sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
REJECT     tcp  --  anywhere             169.254.169.254      owner UID match 1000-10000 reject-with icmp-port-unreachable

If the result of the iptables listing shows a metadata retrieval rejection, then you might receive the following error:

$ curl http://169.254.169.254/latest/meta-data/
curl: (7) Failed to connect to 169.254.169.254 port 80 after 0 ms: Connection refused

To resolve this error, run the following command to delete the rule:

$ sudo iptables -D OUTPUT -proto tcp -destination 169.254.169.254 -match owner -uid-owner 1000-10000 -jump REJECT

Request throttling

Traffic to the IMDS is throttled based on the number of packets per second. And, there is a limit of 1024 PPS for each Elastic Network Interface attached to the instance. If your PPS rate to the IMDS service exceeds 1024 PPS, then the request is throttled.

If throttling occurs, then retry your request with an exponential backoff strategy.

To view how often EC2 instances reach throttling limits, check the linklocal_allowance_exceeded metric in the Elastic Network Interface driver. This metric indicates that the number of packets shaped due to the traffic to local proxy services exceeds the maximum for the network interface.

Run the following command to view the linklocal_allowance_exceeded metric:

$ ethtool -S <Network-Interface>
eg: $ ethtool -S eth0

Related information

Using a proxy on Amazon EC2 instances

Examples of retrieving instance metadata

Limit IMDS access

Query throttling

Metrics for the ENA driver