Getting secret from Lambda times out when attached to VPC subnet

1

When a Lambda is attached to one (or more) VPC subnets, fetching a secret from Secrets Manager with GetSecretValue times out. And when the Lambda is detached from all subnets, the GetSecretValue call succeeds. This happens consistently in both Python 3.6 and Node.js 8.10 Lambda environments, so it's not specific to one language library.

In my use case, I am accessing RDS form Lambda, and as such have attached the Lambda to the VPC. But because of this issue with Secrets Manager, I would need to use something else to store RDS credentials.


EDIT: SOLVED, and will mark it as such when my posting quota (apparently a thing around here) lets me.

According to https://stackoverflow.com/a/35423306,

"Once you enable VPC support in Lambda your function no longer has access to anything outside your VPC. [...] For pretty much anything else outside your VPC, you would need to create a NAT instance or a managed NAT gateway in your VPC to route traffic from your Lambda functions to endpoints outside of your VPC."

mxk
asked 6 years ago12766 views
8 Answers
1

You're 100% correct. When you put your RDS instance and your Lambda in a VPC, you're putting it on a private network that, by default, has no connectivity to the outside world. Secrets Manager's API endpoints are available only on the public Internet, so for your Lambda to run Secrets Manager API operations, you must equip your VPC with some means of accessing the Internet. The easiest way to do that is to add a NAT gateway that allows requests from the VPC to access the Internet and responses to those requests to make their way back.

Dave

AWS
answered 6 years ago
1

Thanks Dave,

I read a lot of recommendations about using a NAT gateway to grant the VPC internet access, but what about this alternative? Going to network interfaces in EC2, finding the ENI labeled "AWS Lambda VPC ENI", and associating a new elastic IP with that interface? I just confirmed this grants internet access to my Lambda, although I have to explicitly associate it with the private subnet of that interface.

This approach isn't mentioned anywhere, and I wonder what its advantages/disadvantages are as compared with using a NAT gateway.

mxk
answered 6 years ago
1

I'd argue that a NAT gateway benefits all of the hosts on the subnet automatically, while an EIP directly affects only the attached host. Also, there is a very significant limit of 5 EIPs per region, so we would not recommend a scheme that results in a 1:1 association between an EIP and a Lambda function. You'd run out of EIPs very quickly.

A NAT also provides some basic filtering that an EIP does not. A NAT essentially filters out all traffic except responses to previous outgoing requests (unless you specifically designate and redirect an incoming port to a specific host).

We recommend that you follow the guidance found in this topic in the Lambda Developers Guide: https://docs.aws.amazon.com/lambda/latest/dg/vpc.html, which very clearly indicates that a NAT gateway is the preferred configuration option.

I hope this helps!
Dave

AWS
answered 6 years ago
1

Hi Dave,

It would be great to have a VPC endpoint for Secrets Manager !

Deploying a NAT Gateway (indeed 3 NAT gtws for 3 stages) only to access Secrets Manager from Lambda is a pity... in particular for cost reasons.

Thanks

Steph
answered 6 years ago
1

Steph,

I'm guessing you haven't seen the latest news. :-)

In July we introduced support for configuring Secrets Manager endpoints within your VPC. For more information, see the announcement blogpost here (https://aws.amazon.com/blogs/security/how-to-connect-to-aws-secrets-manager-service-within-a-virtual-private-cloud/) or the documentation for this feature (https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotation-network-rqmts.html#vpc-endpoint).

That will simplify all this! I hope this helps!
Dave Bishop
Tech Writer for Secrets Manager

AWS
answered 6 years ago
0

Thanks Dave, indeed I didn't see the july announcement ! ;)

Steph
answered 6 years ago
0

Great! Can you please flag my response as the correct answer? Thank you very much!

Dave

AWS
answered 6 years ago
0

Sorry David, it seems that I'm not able to flag your answer..

Steph
answered 6 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions