ec2 assumed role does not work for secret manager

To troubleshoot this issue with one EC2 instance not being able to call Secrets Manager while the other can, despite having the same IAM role, you can try the following steps:

1. Double-check the IAM role permissions: Ensure that the IAM role attached to both EC2 instances has the necessary permissions for Secrets Manager operations, particularly the "secretsmanager:ListSecrets" action.

2. Verify network connectivity: Check if both instances have proper network access to the Secrets Manager service. Ensure that the VPC, subnet, and security group settings allow outbound traffic to AWS services.

3. Compare AWS CLI versions: Make sure both instances are running the same, up-to-date version of the AWS CLI. You can check this by running "aws --version" on both instances.

4. Check for any differences in AWS configuration: Verify that both instances have the same AWS configuration, including region settings. You can check this in the ~/.aws/config file.

5. Examine CloudTrail logs: Look for any API calls or error messages related to Secrets Manager operations from both instances.

6. Monitor system resources: Check if the non-responsive instance has any resource constraints (CPU, memory, disk space) that might be affecting its performance.

7. Test with other AWS services: Try running commands for other AWS services to see if the issue is specific to Secrets Manager or if it's a broader problem with AWS API access.

8. Verify instance metadata: Ensure both instances can access the EC2 instance metadata service by running "curl http://169.254.169.254/latest/meta-data/".

9. Check for any custom configurations or proxy settings that might be interfering with AWS API calls on the problematic instance.

10. If the issue persists, try terminating and relaunching the problematic instance to see if it resolves the problem.

If none of these steps resolve the issue, you may need to contact AWS support for further assistance, as there could be an underlying problem with the instance or the AWS services in your region.
Sources
Possib;e bug with IAM permissions and the secretsmanager:BatchGetSecretValue permission? | AWS re:Post
Troubleshoot Secrets Manager code 400 | AWS re:Post
Resolve Secrets Manager integration issues with MWAA | AWS re:Post

Additional information: I can access other services. Only secrets manager service does not work for that instance.

I added --debug option and the process hung at this line: MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): secretsmanager.[...].amazonaws.com:443.

Initial indication would be that network connectivity is not in place for that destination.
This could be due to networking and security in the VPC or on the instance.
I would take a look at the following:

1. Can you resolve that endpoint dns entry on the box? Use ping or nslookup to do this and make a note of the returned IPs.
2. Are the IPs returned correct? verify against working machine and your local machine.
3. If the IPs are not correct verify host files or ip tables on the instance to see why a different address is returned.
4. Are there any NACLs that would be restricting those IP destinations?
5. Is routing in place for that destination?
6. If routing is via a VPC Endpoint is there a security group or resource policy in place.
7. Is the box running a firewall service that might be blocking the destination? if something like ufw on ubuntu is running temporarily turn it off to validate.

Hope this helps and gets you to see where the issue might be.