AWS Lambda "Internal Server Error"

Question

Hello.
I had an AWS Lambda Function for dotnet 6. One in a staging step that was deployed a few months ago, and now one in a production step that was deployed just yesterday. Both follow the same pipeline, so both are built exactly the same, they just change environment variables that are gotten through AWS Secret Manager. Both deployed in the exact same way, too.
After deploying the same code, the staging one works properly, while the production one gives a `{"message":"Internal Server Error"}`. The Lambda in production has access to a database in RDS that is in a private subnet. I tried seeing if that was causing any issue, but I tried adding the Lambda to the same VPC in the public subnet, and adding a direct connection from Lambda to RDS through the options the Lambda config gives, but neither helped.
I enabled logs, but all I'm getting is:

` [09/Aug/2024:18:17:55 +0000] "POST POST /data HTTP/1.1" 500 35 cQOFFgC6IAMEJFg= The Lambda function returned the following error: 2024-08-09T18:18:20.962Z a52eb4f2-e7a1-4b66-92f6-d985d0c78a0c Task timed out after 15.04 seconds. Check your Lambda function code and try again.`

And the direct logs of the lambda don't show any detail other than it timed out.

randy_weinstein · Answer

Internet access from a private subnet requires either 1) network address translation (NAT) + internet gateway; or 2) Interface Endpoint.

Options:
1) To give internet access to an Amazon VPC-connected Lambda function, route its outbound traffic to a NAT gateway or NAT instance in a public subnet. For an example setup, see [VPC with servers in private subnets and NAT](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html).

2) [Use Interface Endpoints to connect to AWS services from within an Amazon VPC without internet access](https://docs.aws.amazon.com/whitepapers/latest/aws-privatelink/use-case-examples.html).

randy_weinstein · Answer

Given the time-out, I suspect a Security Group issue. I suggest you use Reachability Analyzer to confirm connectivity.

See: [troubleshoot AWS Lambda function connection issues to an Amazon Virtual Private Cloud (Amazon VPC) using VPC Reachability Analyzer](https://repost.aws/knowledge-center/lambda-reachability-analyzer)

The high-level steps are:
1. Get the network interface IDs for Lambda functions
2. Create and analyze a path with Reachability Analyzer
3. View the results of the path analysis

Dmitry_G · Answer

Hi,

I will suggest you to reduce the number of retries and connection timeouts for the AWS SDK client. This will allow actual error to be logged instead of a Lambda function timeout out while re-trying a connection to an unavailable endpoint.

See [Retries and timeouts](https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/retries-timeouts.html) documentation for AWS SDK for .NET.

Once you identify the problematic endpoint, you can focus on fixing connectivity issues with it.

Best regards,

Riku_Kobayashi · Answer

Hello.

You say that Lambda is connected to a VPC, but in that case, is there a route to connect to SecretsManager?  
If you are connecting Lambda to a VPC, you will need to set up a NAT Gateway or VPC endpoint to access each AWS service.  
https://repost.aws/knowledge-center/internet-access-lambda-function

Also, can I check Lambda logs with CloudWatch Logs?  
If a request is received from API Gateway to Lambda, logs will be output to CloudWatch Logs.

AWS Lambda "Internal Server Error"

Relevant content