By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

EC2 instance can't connect to AWS SecretManager via private IP

2

When debugging a startup issue in our ElasticBeanstalk container, my startup was hanging on aws secretsmanager get-secret-value --secret-id=[redacted] --region='us-east-1'. It would hang indefinitely and not give me any "denied" message or any other indication that I had a permission issue in SecretsManager.

I narrowed the issue down to reachability to SecretsManager when I discovered that I couldn't even telnet to SecretsManager via telnet secretsmanager.us-east-1.amazonaws.com 443. It wouldn't even connect.

I then noticed that it was resolving SecretsManager's IP to an IP that is internal to our VPC. Updating /etc/resolv.conf to use another DNS provider switched the IP address and we were able to connect no problem. I am moving forward with a fix that will add a static IP to /etc/hosts.

Interestingly, AWS's Reachability Analyzer reports that my instances should be able to reach secretsmanager.us-east-1.amazonaws.com. I am able to reach other AWS services like s3, etc. I don't have any outbound Security Group settings that should prevent this. Everything in this instance is AWS managed. It's an AWS image with no special configuration on our side. It's a AWS Linux Corretto Java 17 image.

Any thoughts on what may have caused this seemingly out of the blue? Any thoughts on what to probe at to determine where the issue is coming from?

Topics
Networking & Content DeliveryCompute
Tags
Amazon VPCLinux ProvisioningAmazon EC2Security Group
Language
English
mressler
asked 2 years ago224 views
1 Answer
0

Thank you for the detailed description.

resolving SecretsManager's IP to an IP that is internal to our VPC is an indication that you might have deployed Secrets Manager VPC endpoint [1] in your VPC. If this is the case, you might want to review the security group [2], as well as the endpoint policy, associated with this endpoint to make sure that they both allow your container's IP to connect through.

[1] https://docs.aws.amazon.com/secretsmanager/latest/userguide/vpc-endpoint-overview.html

[2] https://docs.aws.amazon.com/vpc/latest/privatelink/interface-endpoints.html#associate-security-groups

AWS
weidi
answered 2 years ago
  • ctucker
    2 years ago

    Hello - I'm a colleague of @mressler - we are utilizing a VPC endpoint for secrets manager. We believe that our instances should have access both via manual checking, and by using AWS's Reachability Analyzer to ensure that there is a successful path from EB instance to Secrets Manager VPC Endpoint. Are there any other suggestions for troubleshooting this issue? Edit: I just tried using port 443 on the analyzer and it is now saying Not reachable - I will investigate this further. Edit2: I've added a rule to allow that traffic through for port 443 over IPV4 and we're now seeing that connection work properly. Though, now I'm confused why the IPV6 rule didn't catch this.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content