Specific exception

Question

Hello,
We are running an app on our server and using IAMRoles Anywhere feature to generate temporary credentials. Temporary credentials are generated using command './aws_signing_helper credential-process --certificate ./certificate.pem --private-key ./privkey.pem --trust-anchor-arn  xx --profile-arn yy --role-arn zz`
The generated temporary credentials are used to upload the content to media store.

Code works fine for sometime, chunks are uploaded into media store. But after sometime (after approx 3-6 mins) i get below exception and uploading stops:
Note: Credentials are not expired.
=================================================
Error: connect ETIMEDOUT 169.254.169.254:80
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1137:16) {
  message: 'Could not load credentials from any providers',
  errno: 'ETIMEDOUT',
  code: 'CredentialsError',
  syscall: 'connect',
  address: '169.254.169.254',
  port: 80,
  time: 2023-07-06T12:18:11.025Z,
  originalError: {
    message: 'EC2 Metadata roleName request returned error',
    errno: 'ETIMEDOUT',
    code: 'ETIMEDOUT',
    syscall: 'connect',
    address: '169.254.169.254',
    port: 80,
    time: 2023-07-06T12:18:11.024Z,
    originalError: {
      errno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '169.254.169.254',
      port: 80,
      message: 'connect ETIMEDOUT 169.254.169.254:80'
    }
  }
}=========================================================
Any idea how to resolve it?

Yusuf Adeyemo · Answer

The error you're experiencing is related to the AWS SDK not being able to connect to the EC2 instance metadata service, which is typically available at `169.254.169.254:80`. This service is used to retrieve temporary credentials for IAM roles.

The `ETIMEDOUT` error indicates that the connection to this service is timing out. This could be due to several reasons such as network issues, high CPU utilization, or the SDK being configured incorrectly.

Here are a few steps you can take to troubleshoot and potentially resolve this issue:

1. **Check Network Connectivity**: Ensure that your server has a stable network connection and can reach `169.254.169.254:80`. You can test this by trying to curl the metadata endpoint from your server: `curl http://169.254.169.254/latest/meta-data/`. If you can't reach the endpoint, you may have a network issue.

2. **Check CPU Utilization**: High CPU utilization can cause the SDK to timeout when trying to connect to the metadata service. Monitor your server's CPU usage to see if it's consistently high.

3. **SDK Configuration**: Ensure that your AWS SDK is configured to use IAM roles for EC2 instances. If it's incorrectly configured to use another method of authentication, it could be timing out when trying to reach the metadata service. You can typically specify this configuration when initializing the SDK.

4. **Increase SDK Timeout**: If the SDK's timeout is set too low, it may not be able to establish a connection to the metadata service before timing out. You can try increasing the timeout value in the SDK configuration.

5. **Update SDK and Dependencies**: Ensure that your AWS SDK and its dependencies are up to date. Older versions may have bugs or compatibility issues that cause this error.

Remember to always test changes in a controlled and isolated environment before applying them to your production environment.

Didier Durand · Answer

Hi, the credentials when you create an auth session have a validity duration. See https://docs.aws.amazon.com/rolesanywhere/latest/userguide/authentication-create-session.html

Did you set the duration for your use case at the right length? 
The message that you get may mean that initial credentials have expired. If it's the case, you should increase the duration

Best,
Didier

Specific exception

Code works fine for sometime, chunks are uploaded into media store. But after sometime (after approx 3-6 mins) i get below exception and uploading stops: Note: Credentials are not expired.

Add your answer

Relevant content