How do I troubleshoot Lambda function invocation timeout errors?

Resolution

Lambda functions can time out for a variety of reasons. To troubleshoot Lambda function timeouts, first determine what caused the issue. Then, remediate the problem based on your use case.

Verify that your Lambda function timed out

To retrieve the request IDs of any timed-out invocations, search the function's Amazon CloudWatch log group for the phrase Task timed out. Then, use the request IDs of the associated timed-out invocations to retrieve the full logs for each invocation timeout.

For instructions, see How do I determine if my Lambda function is timing out?

Identify what's causing your Lambda function to time out

Use one or more of the following methods to identify the point of failure that caused your function to time out:

Review your CloudWatch Logs for Lambda

You can use Amazon CloudWatch to view all logs generated by your function's code and identify potential issues. For instructions, see Using CloudWatch Logs with Lambda.

If your function returned a stack trace, then the error message in the stack trace specifies what caused the error.

Important: Lambda generates three log lines for each invocation automatically (START, END, and REPORT). These three lines are the only lines that appear in your function's CloudWatch logs if either of the following is true:

• There's no other explicit logging configured in the Lambda function's custom code.
• The function's duration limit is reached before Lambda can run the function's code that outputs logs.

If you can't determine what caused the timeouts from the CloudWatch logs, then try one or more of the following solutions:

• Verify that the retry count and timeout settings on the AWS SDK that you used allow enough time for your function to initialize.
• Increase the Lambda function's timeout setting temporarily to allow enough time for the function code to generate log data.
• Increase the function's configured memory to help reduce invocation duration latency and increase computing power.

To add more logging output to your function's code, see the following documentation for the Lambda runtime version that you used:

• AWS Lambda function logging in Node.js
• AWS Lambda function logging in Python
• AWS Lambda function logging in Ruby
• AWS Lambda function logging in Java
• AWS Lambda function logging in Go
• AWS Lambda function logging in C#
• AWS Lambda function logging in PowerShell

Use AWS X-Ray to identify any code performance bottlenecks

If your Lambda function uses downstream AWS resources, microservices, databases, or HTTP web APIs, then you can use AWS X-Ray to help troubleshoot code performance issues.

For more information, see Visualize Lambda function invocations using AWS X-Ray.

Use Lambda Insights to collect system-level metrics for your function

Lambda Insights collects system-level metrics, which include CPU time, memory, disk, and network metrics. It also collects diagnostic information, which includes cold starts and Lambda worker shutdowns to help you isolate issues with your Lambda functions.

Note: Use of Lambda Insights incurs charges to your AWS account. You're charged for the invocation time consumed by the Lambda extension in 1 ms increments.

Use VPC Flow Logs to determine why a specific invocation request was denied or didn't route

VPC Flow Logs allow you to see all the network traffic that flows to and from an Amazon Virtual Private Cloud (Amazon VPC).

For more information, see Troubleshoot networking issues in Lambda.

Note: The following variables apply if you choose to set up VPC Flow Logs:

• Data ingestion and archival charges for vended logs apply when you publish flow logs to CloudWatch Logs or Amazon Simple Storage Service (Amazon S3).
• When you configure a Lambda function to access resources in an Amazon VPC, Lambda assigns the function to an elastic network interface. To identify the network traffic associated with your Lambda function, you must locate the function's network interface. For instructions, see Why can't I detach or delete an elastic network interface that Lambda created?

Use HTTP wire traces for detailed logging of network requests that are generated by the function's code during an invocation

For more information, see Logging HTTP wire traces.

Best practices to prevent Lambda function timeouts

Make sure that your Lambda function is idempotent

API calls might take longer than expected due to transient network issues. Network issues can also cause retries and duplicated API requests. To prepare for these occurrences, make sure that your Lambda function is idempotent.

For more information, see How do I make my Lambda function idempotent?

Initialize your function's static logic outside of the function handler

When you initialize a Lambda function, Lambda allocates up to 10 seconds for the Init phase of the invocation to complete. Because of this time constraint, it's a best practice to perform the following actions outside of the function handler in initialization code:

• Import libraries and dependencies
• Set up the initial configuration
• Initialize connections to other downstream services and resources

This static initialization allows these resources to be initialized once per sandbox and then reused for all future invocations in the same runtime environment.

For more information, see Optimizing static initialization.

Note: Lambda removes idle connections to downstream resources. To allow your function to maintain a persistent connection, use the TCP keepAlive variable that's associated with the Lambda runtime that you use.

Verify that the retry count and timeout settings on the AWS SDK that you use allows enough time for your function to initialize

If you used an AWS SDK to make an API call and the call fails, then the AWS SDK automatically retries the call. The number of times the AWS SDK retries and the length of each retry is determined by settings that vary between each AWS SDK. Your function might need more time to initialize than the default AWS SDK settings allow.

For more information, see How do I troubleshoot retry and timeout issues when invoking a Lambda function using an AWS SDK?

(Optional) Configure provisioned concurrency for your Lambda function

Provisioned concurrency initializes a requested number of runtime environments so that they're prepared to respond immediately to your function's invocations. To set up provisioned concurrency for your function, follow the instructions in Configuring provisioned concurrency.

Note: Configuring provisioned concurrency incurs charges to your AWS account. You can configure provisioned concurrency on a version of a function or on a Lambda function alias.

Verify that your Lambda function has enough system resources

The amount of network bandwidth and CPU allocated to a Lambda function invocation is determined by the function's memory configuration.

For more information, see Memory and computing power.

Make sure that any background processes your Lambda function uses complete before the function handler returns a string

For more information, see Understanding container reuse in AWS Lambda.

Verify that your Lambda function is configured to work within the maximum timeout settings of any integrated AWS services

Even though a Lambda function's maximum invocation timeout limit is 15 minutes, other AWS services might have different timeout limits.

For example, Amazon API Gateway waits a maximum of 29 seconds for a Lambda function proxy invocation to complete.

For more information, see How can I resolve the errors that I receive when I integrate API Gateway with a Lambda function? Also, see Invoking Lambda with events from other AWS services.

Confirm that there's a valid network path to the endpoint that your function is trying to reach

To review your network settings, follow the instructions in How do I troubleshoot timeout issues with a Lambda function that's in an Amazon VPC?