How do I troubleshoot Lambda function "ETIMEDOUT" errors?

Short description

You received the "ETIMEDOUT" error because the host didn't respond before the connection timed out. Network configuration issues can cause an incorrect network path between the Lambda function and the host. This incorrect path causes connection timeout errors. To troubleshoot an "ETIMEDOUT" error, check your networking configuration based on your host configuration.

Note: If you receive the error "ENOTFOUND," "getaddrinfo," or "UNKNOWNHOSTEXCEPTION" instead of "ETIMEDOUT," then the issue is DNS related. For more information, see How do I troubleshoot Lambda DNS related errors?

Resolution

Before you begin to troubleshoot, make sure that your HTTP client timeout setting is at least 30 seconds to rule out client-side issues.

Note: If the "ETIMEDOUT" error occurs after a NAT gateway connection is idle for 350 seconds or more, then check your configuration requirements. Turn on TCP keepalive with a value of less than 350 seconds to reset the idle timeout. For more information, see How can I prevent my Lambda function connected to an Amazon VPC from timing out?

For a host that's accessible through the public internet

If you configured the function with Amazon Virtual Private Cloud (Amazon VPC), then make sure that the function has access to the public internet. If you receive intermittent timeout errors, then confirm that all subnets configured for the function are private and have default routes to a NAT Gateway. For default routes to an AWS Transit Gateway transit gateway, confirm that the transit gateway allows the function to access the public internet.

Note: By default, a Lambda function without VPC configurations has access to the public internet. If you haven't configured your function with Amazon VPC and you receive "ETIMEDOUT" errors, then check your configuration requirements. The destination host is unresponsive because of rate-limited requests or blocked Lambda IP ranges. Use exponential backoff to reduce your service load and increase request success. For more information, see Retry with backoff pattern.

For functions configured with Amazon VPC, confirm your network settings. Take the following actions:

• The NAT Gateway is in an Available state and placed in a public subnet with a route to an internet gateway.
• Because Lambda uses multiple subnets for high availability, make sure that every subnet configured for the function has a route to a NAT Gateway.
• If your subnet uses a network ACL, then allow the full ephemeral port range 1024–65535 for both TCP and UDP. Because Lambda Hyperplane Elastic Network Interfaces (ENIs) use port range 1024–65535, restrictions cause intermittent connection failures.
• Make sure that your Lambda function doesn't receive incoming fragmented TCP requests. If your destination host returns responses that cause IP fragmentation, then you receive "ETIMEDOUT" errors.

For a host that's accessible only within a VPC

Configure the Lambda function in the VPC and then confirm your network settings. Take the following actions:

• Make sure that the route table has a valid path to the host. If the host endpoint is in the same VPC as the Lambda function, then check that there's a local route. If the host endpoint is in a different VPC, then make sure that the route table has the route for the VPC peering connection. Also, make sure that you activated DNS resolution on both the requester and accepter sides.
• The subnet's network access control list (network ACL) rules in both the Lambda function and host allow inbound and outbound rules.
• The Lambda function's security group rules allow outbound traffic to the destination host on the required port.
• The host's security group rules allow inbound traffic from Lambda function on the required port.
• The host supports requests with the HTTP or HTTPS protocol and ports specified in the connection request.
• If your function accesses AWS services from a private subnet without internet access, then check your configuration requirements. Confirm that the interface VPC endpoint has private DNS activated and its security group allows inbound HTTPS traffic from the Lambda security group. Also, make sure that the endpoint policy allows the required actions.

To analyze network path configuration issues, use the VPC Reachability Analyzer.

For a host that uses a VPC with a custom DHCP option set

If the VPC has a custom Dynamic Host Configuration Protocol (DHCP) option set, then check your configuration requirements. Use Amazon Route 53 Resolver Query Logs to view the DNS query response. The host's IP address must be can connect to the port. Also, the host must use a private Amazon Elastic Compute Cloud (Amazon EC2) instance that's in the same subnet and security group as the function.

To check the host's IP address, use the Amazon EC2 instance to run the following command:

nslookup domain.com

Note: Replace domain.com with your host name.

To verify that the IP address can connect to the requested port, run the following command:

telnet host-name-IP-address 443 

Note: Replace host-name-IP-address with your host name or with your IP address and 443 with your port.

For more information, see How do I troubleshoot Lambda DNS related errors?

Check your host configurations

Confirm whether the downstream service or the external endpoint received the request. Use the host's available metrics, logs, or network traces. If you use a VPC, then use VPC Flow Logs to confirm that the outbound traffic left the Lambda function. If you use an elastic network interface, then, use the Lambda ENI Finder on the GitHub website to find the IP address. For more information, see Why can't I detach or delete an elastic network interface that Lambda created?

If high incoming traffic blocks the host so that it can't respond within the configured socket timeout, then use the retry with back off pattern. For persistent intermittent failures, implement a circuit breaker pattern to temporarily stop requests after consecutive failures.

Related information

How do I troubleshoot Lambda DNS related errors?

Troubleshoot networking issues in Lambda

Giving Lambda functions access to resources in an Amazon VPC

How do I use Amazon VPC Reachability Analyzer to troubleshoot connectivity issues with an Amazon VPC resource?