When debugging a startup issue in our ElasticBeanstalk container, my startup was hanging on aws secretsmanager get-secret-value --secret-id=[redacted] --region='us-east-1'. It would hang indefinitely and not give me any "denied" message or any other indication that I had a permission issue in SecretsManager.
I narrowed the issue down to reachability to SecretsManager when I discovered that I couldn't even telnet to SecretsManager via telnet secretsmanager.us-east-1.amazonaws.com 443. It wouldn't even connect.
I then noticed that it was resolving SecretsManager's IP to an IP that is internal to our VPC. Updating /etc/resolv.conf to use another DNS provider switched the IP address and we were able to connect no problem. I am moving forward with a fix that will add a static IP to /etc/hosts.
Interestingly, AWS's Reachability Analyzer reports that my instances should be able to reach secretsmanager.us-east-1.amazonaws.com. I am able to reach other AWS services like s3, etc. I don't have any outbound Security Group settings that should prevent this. Everything in this instance is AWS managed. It's an AWS image with no special configuration on our side. It's a AWS Linux Corretto Java 17 image.
Any thoughts on what may have caused this seemingly out of the blue? Any thoughts on what to probe at to determine where the issue is coming from?
AWS 官方已更新 2 年前
Hello - I'm a colleague of @mressler - we are utilizing a VPC endpoint for secrets manager. We believe that our instances should have access both via manual checking, and by using AWS's Reachability Analyzer to ensure that there is a successful path from EB instance to Secrets Manager VPC Endpoint. Are there any other suggestions for troubleshooting this issue? Edit: I just tried using port 443 on the analyzer and it is now saying Not reachable - I will investigate this further. Edit2: I've added a rule to allow that traffic through for port 443 over IPV4 and we're now seeing that connection work properly. Though, now I'm confused why the IPV6 rule didn't catch this.