How do I resolve the "The security token included in the request is expired" error that occurs when I run Java applications on Amazon EC2?

Short description

All application API requests must be digitally signed with credentials that AWS provided. If your application uses temporary credentials to create an AWS client, then you must renew these credentials before they expire. If your credentials expire, then you receive the "security token included in the request is expired" error message.

Resolution

Make sure that your instance has a consistent time reference

Credentials expire if they use the incorrect time, so make sure that your server is accurate. Your EC2 instance must have a consistent and accurate time and date reference. Configure the Amazon Time Sync Service or another Network Time Protocol (NTP) source on your instance. For more information, see Change the time zone of your instance.

Make sure that your temporary credentials aren't expired

If you use temporary credentials, then make sure that they're not expired. If the temporary credentials are expired, then you must generate a new set of temporary credentials and use those. It's a best practice to refresh temporary credentials five minutes before they expire.

Check your IAM configuration

For applications that run on an instance, it's a best practice to use an AWS Identity and Access Management (IAM) role for the instance.

If you use an IAM role, then make sure to list the role's credentials only in the instance profile. With an IAM role, the default service constructor uses a default credentials provider chain to search for credentials in the following order:

1. In the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY system environment variables.
2. In the aws.accessKeyId and aws.secretKey Java system properties.
3. In the default credentials file.
4. In the instance profile credentials in the instance metadata that's associated with the IAM role.

If you list credentials anywhere except the instance profile, then the default client constructor finds those credentials first. This configuration blocks the IAM role credentials. To resolve this issue, check the other locations for credentials. To check the default credential file, check the following location based on your operating system (OS):

• For Linux, macOS, or Unix, check * ~/.aws/credentials.
• For Windows, check C:\Users\USERNAME\.aws\credentials
  Note: Replace USERNAME with your username.

If you find credentials anywhere except the instance profile, then remove them. For more information about temporary credentials configurations, see Provide temporary credentials to the AWS SDK for Java.

To check the credentials for the IAM role, run the following command based on your OS. Make sure to use Windows PowerShell version 3.0 or later or from a Linux shell.

Windows:

PS C:\> Invoke-RestMethod http://169.254.169.254/latest/meta-data/iam/security-credentials/examplerole

Note: Replace examplerole with the name of your IAM role. If you use temporary credentials, then the preceding command shows the latest temporary credentials for the instance.

Example output:

Code            : SuccessLastUpdated     : 2016-07-18T18:09:47Z
Type            : AWS-HMAC
AccessKeyId     : AKIAIOSFODNN7EXAMPLE
SecretAccessKey : wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Token           : token
Expiration      : 2016-04-27T22:39:16Z

Linux:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/examplerole

Note: Replace examplerole with the name of your IAM role. If you use temporary credentials, then the preceding command shows the latest temporary credentials for the instance. If you receive a 404 error when you run the preceding command, then verify that you deactivated the HTTP proxy for the metadata IP address. Also, make sure that you attached the instance profile to the instance.

Example output:

{    "Code" : "Success",    "LastUpdated" : "2016-04-26T16:39:16Z",
    "Type" : "AWS-HMAC",
    "AccessKeyId" : "AKIAIOSFODNN7EXAMPLE",
    "SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "Token" : "token",
    "Expiration" : "2016-04-27T22:39:16Z"
}

Note: Role credentials automatically rotate or refresh five minutes before the assigned temporary credentials expire.

Configure retries

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

If you still encounter issues, then verify that the instance isn't making multiple concurrent requests or running multiple sessions in parallel. This scenario can cause the Instance Metadata Service (IMDS) to throttle the query. To mitigate this issue, use the retry with exponential backoff pattern.

To set up retries, increase the value of AWS_METADATA_SERVICE_NUM_ATTEMPTS so that Boto3 can retry attempts to get credentials from the instance metadata service. By default, Boto3 only makes one attempt before it times out. Also, increase the value of AWS_METADATA_SERVICE_TIMEOUT to increase the number of seconds before the connection to the instance metadata service times out. By default, the connection times out after 1 second.

Example configuration:

AWS_METADATA_SERVICE_TIMEOUT = 10
AWS_METADATA_SERVICE_NUM_ATTEMPTS = 5

To configure the preceding options, use environment variables, the ~/.aws/config file, or the user's botocore session. For instructions, see Configuration on the Boto3 documentation website.

If you run the curl command in a Docker container, then run the following modify-instance-metadata-options AWS CLI command to increase the hop limit to 2:

aws ec2 modify-instance-metadata-options --instance-id instance --http-put-response-hop-limit 2 --http-endpoint enabled

Note: Replace instance with your instance ID. For more information, see In a container environment, consider reconfiguration or increasing the hop limit to 2 on Instance metadata access considerations.

Related information

Authenticating with AWS using the AWS SDK for Java 2.x

IAM roles for Amazon EC2

Add defense in depth against open firewalls, reverse proxies, and SSRF vulnerabilities with enhancements to the Amazon EC2 Instance Metadata Service