How do I troubleshoot Lambda function invocation timeout errors?

Resolution

To troubleshoot Lambda function timeouts, first determine what caused the issue. Then, remediate the problem based on your scenario.

Verify that your Lambda function timed out

To retrieve the request IDs of timed-out invocations, search the function's Amazon CloudWatch log group for Status: timeout in the REPORT line. For functions that use the legacy log format, search for Task timed out. Then, use the request IDs of the associated timed-out invocations to retrieve the full logs for each invocation timeout.

Note: If your existing CloudWatch Logs filters or metric filters use the legacy format, then update them to match the updated and legacy patterns:

filter @message like /Status: timeout/ or @message like /Task timed out/

For instructions, see How do I use CloudWatch Logs to determine if my Lambda function timed out?

Identify what caused your Lambda function to time out

Use one or more of the following methods to identify the point of failure that caused your function to time out:

Review your CloudWatch Logs for Lambda

Use Amazon CloudWatch to view all logs generated by your function's code and identify potential issues. For instructions, see Sending Lambda function logs to CloudWatch Logs.

If your function returned a stack trace, then the error message in the stack trace specifies what caused the error.

Important: Lambda generates three log lines for each invocation automatically (START, END, and REPORT). These three lines are the only lines that appear in your function's CloudWatch logs if either of the following is true:

• There's no other explicit logging configured in the Lambda function's custom code.
• The function's duration limit is reached before Lambda can run the function's code that outputs logs.

If you can't determine what caused the timeouts from the CloudWatch logs, then try one or more of the following solutions:

• Confirm that the retry count and timeout settings on the AWS SDK that you used allow enough time for your function to initialize.
• Increase the Lambda function's timeout setting temporarily to allow enough time for the function code to generate log data.
• Increase the function's configured memory to help reduce invocation duration latency and increase computing power.

To add more logging output to your function's code, see the following documentation for the Lambda runtime version that you used:

• AWS Lambda function logging in Node.js
• AWS Lambda function logging in Python
• AWS Lambda function logging in Ruby
• AWS Lambda function logging in Java
• AWS Lambda function logging in Go
• AWS Lambda function logging in C#
• AWS Lambda function logging in PowerShell

Use AWS X-Ray to identify any code performance bottlenecks

If your Lambda function uses downstream AWS resources, microservices, databases, or HTTP web APIs, then you can use AWS X-Ray to help troubleshoot code performance issues.

For more information, see Visualize Lambda function invocations using AWS X-Ray.

Use Lambda Insights to collect system-level metrics for your function

Lambda Insights collects system-level metrics, which include CPU time, memory, disk, and network metrics. It also collects diagnostic information, which includes cold starts and Lambda worker shutdowns to help you isolate issues with your Lambda functions.

Note: Use of Lambda Insights incurs charges to your AWS account. You're charged for the invocation time consumed by the Lambda extension in 1 ms increments.

Use VPC Flow Logs to determine why a specific invocation request was denied or didn't route

VPC Flow Logs allow you to see all the network traffic that flows to and from an Amazon Virtual Private Cloud (Amazon VPC).

For more information, see Troubleshoot networking issues in Lambda.

Note: The following variables apply if you choose to set up VPC Flow Logs:

• Data ingestion and archival charges for vended logs apply when you publish flow logs to CloudWatch Logs or Amazon Simple Storage Service (Amazon S3).
• When you configure a Lambda function to access resources in an Amazon VPC, Lambda assigns the function to an elastic network interface. To identify the network traffic associated with your Lambda function, you must locate the function's network interface. For instructions, see Why can't I detach or delete an elastic network interface that Lambda created?

Use HTTP wire traces for detailed logging of network requests that are generated by the function's code during an invocation

For more information, see Logging HTTP wire traces.

Best practices to prevent Lambda function timeouts

Make sure that your Lambda function is idempotent

API calls might take longer than expected because of transient network issues. Network issues can also cause retries and duplicated API requests. To prepare for these occurrences, make sure that your Lambda function is idempotent.

For more information, see How do I prevent duplicate Lambda events?

Initialize your function's static logic outside of the function handler

When you initialize a Lambda function, Lambda allocates up to 10 seconds for the Init phase of the invocation to complete. Because of this time constraint, it's a best practice to perform the following actions outside of the function handler in initialization code:

• Import libraries and dependencies
• Set up the initial configuration
• Initialize connections to other downstream services and resources

This static initialization allows these resources to be initialized after per sandbox and then reused for all future invocations in the same runtime environment.

For more information, see Optimizing static initialization.

Note: Lambda removes idle connections to downstream resources. To allow your function to maintain a persistent connection, use the TCP keepAlive variable that's associated with the Lambda runtime that you use. If your function times out during the init phase before it reaches the handler code, then see How do I troubleshoot Lambda cold start issues?

Verify that the retry count and timeout settings on the AWS SDK that you use allows enough time for your function to initialize

If you used an AWS SDK to make an API call and the call fails, then the AWS SDK automatically retries the call. The number of times the AWS SDK retries and the length of each retry is determined by settings that vary between each AWS SDK. Your function might require more time to initialize than the default AWS SDK settings allow.

For more information, see, How do I troubleshoot retry and timeout issues when I use an AWS SDK to invoke a Lambda function?

(Optional) Configure provisioned concurrency for your Lambda function

Provisioned concurrency initializes a requested number of runtime environments so that they're prepared to respond immediately to your function's invocations. To set up provisioned concurrency for your function, see Configuring provisioned concurrency.

Note: Provisioned concurrency incurs charges to your AWS account. You can configure provisioned concurrency on a version of a function or on a Lambda function alias.

Verify that your Lambda function has enough system resources

The amount of network bandwidth and CPU allocated to a Lambda function invocation is determined by the function's memory configuration.

For more information, see Determining the appropriate memory setting for a Lambda function.

Make sure that any background processes your Lambda function uses complete before the function handler returns a string

For more information, see Understanding container reuse in AWS Lambda.

Verify that your Lambda function is configured to work within the maximum timeout settings of any integrated AWS services

Even though a Lambda function's maximum invocation timeout limit is 15 minutes, other AWS services might have different timeout limits.

For example, API Gateway REST APIs have a 29 second default integration timeout. For Regional and private REST APIs, you can request an increase up to 90 seconds. If your function exceeds the configured timeout, then API Gateway returns a 504 error. For long running operations, implement an asynchronous pattern or use Step Functions.

For more information, see Invoking Lambda with events from other AWS services.

Confirm that there's a valid network path to the endpoint that your function is trying to reach

To review your network settings, follow the instructions in How do I troubleshoot Lambda function "ETIMEDOUT" errors?