EC2 instance metadat credentials initial delay

Question

We make use of EC2 instance profiles to grant iam rights to our instances. This works fine. However when we use the cli or powershell within the instance the first call to an AWS service has a long delay, frequently around 30 seconds. Subsequent calls to other APIs are almost instant.

Does anyone else see a delay on initial credentials retrieval? If not, what kind of response do you get for the first call?

The delay is not specific to any AWS API. It could be practically any AWS cli command.

Any ideas on how I could reduce this delay or how I could troubleshoot further to find the specific call?

Many thanks

Answer

There are a couple of things I would check:

* Make sure you're using the latest version of the AWS CLI
* Verify this only happens with calls out to the AWS API and not any other network calls (i.e. via curl)
* Monitor your CloudTrail for the target account for failed login attempts. It could be that the CLI is trying to authenticate with a stale set of credentials stored in the credentials file or an environment variable, and falling back to the IAM profile role.
* Monitor your VPC Flow Logs to see exactly where/when the slow-down is occurring.
* Ensure proper configuration and routing to your NAT gateways and/or transit networks

EC2 instance metadat credentials initial delay

Relevant content