Skip to content

Lambda suddenly experiencing timeouts

0

Hello,

I am looking some recommendations regarding Lambda timeouts that started happening in the last 3-4 days. This system has been running with no issues for over 3 years, it consists of an API gateway in front of a lambda function which essentially just saves a small entry in dynamodb, it typically executes in less than 300ms, but in the last few days it started timing out randomly. It does not matter what the timeout limit is, sometimes it will execute in 300ms, and then another time executing it with the same parameters takes over 30 seconds (or whatever the timeout is set to) then fails. Any idea why this would suddenly occur after > 3 years of no issues? The system has not been touched in the last 6 months or so.\

Any ideas on what could be happening? Any help is appreciated, thank you.

2 Answers
0

Based on your description, there are several potential causes for the sudden timeouts in your Lambda function after years of stable operation:

  1. Network Configuration Issues: The ETIMEDOUT error can occur when there's a connection failure because the host didn't respond within a specific period. This could indicate a network path issue between your Lambda function and DynamoDB.

  2. Downstream Service Problems: DynamoDB might be experiencing intermittent issues or throttling. Even though your function is simple, if DynamoDB is not responding in a timely manner, your function will time out.

  3. Cold Start Issues: If your function's invocation pattern has changed recently, you might be experiencing more cold starts than before, which could lead to timeouts during the initialization phase.

  4. Resource Constraints: Your Lambda function might not have enough memory or CPU allocated to handle the current workload efficiently.

  5. VPC Configuration: If your Lambda is in a VPC, check that all subnets have valid routes to NAT Gateways or Transit Gateways, and that network ACLs and security groups are properly configured.

To troubleshoot this issue, I recommend:

  • Increase the function's memory allocation, which also increases proportional CPU power
  • Add detailed logging to pinpoint where exactly the timeout is occurring
  • Use AWS X-Ray to trace requests and identify bottlenecks
  • Monitor CloudWatch metrics for your Lambda function and DynamoDB to identify any resource constraints or unusual patterns
  • Check if there have been any AWS service issues in your region during the timeouts
  • Implement proper error handling and retry logic in your function to handle temporary failures

Since this is a sudden change after years of stability, it's likely that something in the environment has changed, rather than an issue with your code. It could be related to AWS infrastructure changes, increased load, or changes in how DynamoDB is responding to your requests.
Sources
Troubleshoot ETIMEDOUT error in my Lambda function | AWS re:Post
Lambda Timeout Errors | AWS re:Post
Troubleshoot invocation issues in Lambda - AWS Lambda

answered 10 months ago

  • To specify:

    • Cold start times are not long at all
    • 1024 MB is set for the lambda memory, it only reports to use 120MB max
    • The lambda is not in a VPC
    • There is no error coming from the lambda, just normal logs, even right up until the end
    • Sometimes nothing in the lambda runs, sometimes everything executes, all the way until the very end (at the return statement) then it times out
0

Why Lambda Suddenly Started Timing Out

For a Lambda → DynamoDB → API Gateway flow that has worked for years, intermittent timeouts are almost always caused by one of these:

1. Hidden DynamoDB Latency / Auto-Retries

Even small, occasional throttles or network jitter cause the AWS SDK to quietly retry in the background. This can push execution time from 300ms → 30s unexpectedly.

Trigger patterns:

  • Hot partition / repeatedly writing to same key
  • Increased traffic on table or GSI
  • DynamoDB p95/p99 latency spikes

2. API Gateway Integration Timeout Mismatch

API Gateway has a ~29s hard limit. If Lambda finishes after API Gateway's timeout, you will see:

  • Lambda logs show "finished normally"
  • API Gateway returns timeout

Check the Integration Timeout on API Gateway.

3. Lambda Returns but Event Loop Is Still Busy (Node/Python)

If your code leaves:

  • open sockets
  • long-lived connections
  • un-awaited promises / background tasks

→ Lambda appears to “complete,” but still runs until timeout.

Common when database clients are recreated inside handler or not properly reused.

4. Occasional DNS / TLS Connection Stall to DynamoDB

Rare but real; first request sometimes blocks → SDK retries → timeout.

Fixable with shorter SDK timeouts and AbortController/Config(read_timeout=...).


How to Diagnose Quickly (15–20 minutes)

StepActionInterpretation
1Invoke Lambda directly (no API Gateway)If still slow → problem is Lambda/DynamoDB.
2Enable X-Ray for Lambda + API GWLook for where the time is spent: handler vs DynamoDB.
3Add log timestamps & getRemainingTimeInMillis() before/after the DynamoDB callShows if the slowdown is in DynamoDB or after handler finish.
4Check DynamoDB metrics: ThrottledRequests, p95 Write latencySpikes = cause.
5Check API Gateway Integration TimeoutMust be < Lambda timeout.

Quick Fixes Based on Diagnosis

If Diagnosis Shows…Do This
DynamoDB call is slowReduce retries + add request timeouts; check hot partition patterns; increase WCU/On-Demand capacity.
Lambda returns but still times outEnsure single shared DynamoDB client outside handler; ensure all Promises awaited; set callbackWaitsForEmptyEventLoop=false if Node.
API Gateway times out firstIncrease API GW timeout or reduce Lambda timeout; keep Lambda ≤ 90% of API GW timeout.
Occasional network stallsAdd SDK timeouts (read_timeout, connect_timeout) and fail fast logic.

Minimum Logging to Add Immediately

Inside handler:

console.log("start", context.getRemainingTimeInMillis());
await ddb.put(...);
console.log("after ddb", context.getRemainingTimeInMillis());
return response;

Look for which log gap consumes time.


What Information Would Help Diagnose Faster

Share these five items (sanitized is fine/preferred):

  1. Language + runtime (Node, Python, etc.)

  2. The DynamoDB access pattern (what is your partition key?)

  3. Lambda CloudWatch “REPORT” logs for:

    • fast execution
    • slow (timeout) execution
  4. DynamoDB CloudWatch metrics (screenshots):

    • ThrottledRequests
    • SuccessfulRequestLatency p95/p99
  5. Your API Gateway Integration Timeout setting

With these, I should be able to pinpoint the root cause precisely and provide more exact fix (single code change or configuration adjustment).


If any part of this explanation was helpful in diagnosing the issue, feel free to let me know — happy to help further.

answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.