AWS Lambda outbound connection timeout

Question

Our team has created a Lambda Function URL to manage background tasks in a queuing fashion. The Lambda essentially schedules the kick off of these tasks by simply calling a RESTful API request to and external endpoint still hosted on AWS. However, we are getting outbound connection timeout to the external endpoint (and basically anywhere) time to time and then unable to send outbound requests for a few hours, eg: after sending 35 requests in 2 hours we are unable to connect for about 3hours. I suspect this being caused by some sort of limit or quota being reached within VPC configuration but I cannot make any sense of this as we don't get any meaningful warning or error. Removing the VPC does solve the timeout problem.

A bit more details about the setup: Lambda has got an RDS attached where it stores the state and updates of last runs and the Lambda is configured with VPC to get access to the RDS instance with the outbound rules such that allows all protocol on all ports. The frequency of the outbound API calls is 5mins with a very light payload, which normally gets a response within 50ms before timeout happens.

What are we doing wrong with the VPC config and shouldn't we get some warning/emails when quotas/limit are reached?

Thanks

Accepted Answer

It's odd that this works and then doesn't; then works again.

From your question I want to clarify: It sounds like the Lambda function is attached to a VPC - so I'm going to go with that assumption.

When you associate your Lambda function with a VPC you nominate the subnets that it should be attached to. From time to time your Lambda function may run in a different back-end context (i.e. different physical server; maybe even a different Availability Zone) and at that time it will use a different subnet.

Is it possible that your subnets are not created "equal"? As in, there's a subnet that's working correctly; but another that isn't? It would be easy to test: Start an EC2 instance in each subnet and try the REST API call to see what result you are getting.

If all is well, I'd suggest a creating a support case to troubleshoot more.

Answer

Hi,

So the issue was indeed with the network configuration of the VPC.  Th VPC we have for the AWS Lambda has 3 availability zones and the Subnets was configured by adding both the 3 private and 3 public subnets, which as I understand is wrong since it's the NAT which translates the private IPs and there is no need to add the public subnets (the Lambda Function only connects to the RDS instance). 
Thank you for your  help, really appreciated!

Answer

If your function is attached to a VPC as implied by your question, make sure that all subnets the function is attached to has a route the NAT Gateway for 0.0.0.0/0.

AWS Lambda outbound connection timeout

Relevant content