How can I prevent my Lambda function connected to an Amazon VPC from timing out?

Lesedauer: 3 Minute
0

I want to prevent my AWS Lambda function connection to an Amazon Virtual Private Cloud (Amazon VPC) from timing out.

Short description

Invoking Lambda function API calls behind a load balancer or NAT gateway without a response might be due to a connection idle timeout issue. The load balancer idle timeout value is 350 seconds for TCP flows.

If the API call response delay is intermittent or less than 350 seconds, this might happen due to retry and timeout issues. For more information, see How do I troubleshoot retry and timeout issues when invoking a Lambda function using an AWS SDK?

Resolution

To prevent the connection from dropping, you can turn on TCP keepalive with a value of less than 350 seconds to reset the idle timeout. TCP keepalive prevents the connection from being idle by sending packets to the front-end and backend connections every 20 seconds.

You can call setKeepAlive on the socket object similar to the following:

Node.js 14 and Node.js 16

const http = require('http');

function invokeApi() {

  return new Promise(resolve => {
   
    var callback = function(response) {
      var str = '';
    
      response.on('data', function (chunk) {
        str += chunk;
      });
     
      response.on('end', function () {
        response = '**** ALB response: ' + str;
        resolve(response)
      });
    }

    const url = 'http://<alb-dns>';
    const myTimeout = 1000*60*15; // 15 minutes
    let req = http.get(url, callback);

    req.on('error', (e) => {
      console.error(`Got error: ${e.message}`);
    });

    req.on('socket', function (socket) {
      socket.setKeepAlive(true, 1000); //set SO_KEEPALIVE
   
      socket.setTimeout(myTimeout);  

      socket.on('timeout', function() {
        console.log('request is about to timeout');
        req.abort();
      });
    });
    req.end();
  })
}

exports.handler = async (event) => {
    let output = await invokeApi();
    const response = {
        statusCode: 200,
        body: output,
    };
    return response;
};

Node.js 18

import * as http from "http"

function invokeApi() {

  return new Promise(resolve => {

    var callback = function(response) {
      var str = '';

      response.on('data', function (chunk) {
        str += chunk;
      });

      response.on('end', function () {
        response = '**** ALB response: ' + str;
        resolve(response);
      });
    }

    const url = 'http://<alb-dns>';
    const myTimeout = 1000*60*15; // 15 minutes
    let req = http.get(url, callback);

    req.on('error', (e) => {
      console.error(`Got error: ${e.message}`);
    });

    req.on('socket', function (socket) {
      socket.setKeepAlive(true, 1000); //set SO_KEEPALIVE

      socket.setTimeout(myTimeout);  

      socket.on('timeout', function() {
        console.log('request is about to timeout');
        req.abort();
      });
    });
    req.end();
  })
}

export const handler = async(event) => {
    let output = await invokeApi();
    const response = {
        statusCode: 200,
        body: output,
    };
    return response;
};

Python 3.x

import json
import socket
import requests
from urllib3.connection import HTTPConnection

HTTPConnection.default_socket_options = HTTPConnection.default_socket_options + [(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)]
def lambda_handler(event, context):
    try:
        r = requests.get("http://<alb-dns>",timeout=(5,900))

        output = r.text
    except requests.RequestException as e:
        print(e)
        raise e
    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": output,
        }),
    }

Using AWS SDK for Java2, you can pass a pre-configured HTTP client similar to the following:

LambdaClient awsLambda = LambdaClient.builder()
    .region(region
    .httpClient(ApacheHttpClient.builder()
                   .tcpKeepAlive(true)
                   .connectionTimeout(Duration.ofSeconds(5))
                   .connectionMaxIdleTime(Duration.ofMinutes(15))
                   .socketTimeout(Duration.ofMinutes(15)).build())
    .build();

Using AWS SDK for Python (Boto3) allows you to use a configuration file. Create a .config file with following contents:

[default]
tcp_keepalive=true

Then, create a Lambda environment variable AWS_CONFIG_FILE with the value of the .config file in the directory /var/task/config.

For more information, see Connection idle timeout.


Related information

Announcing improved VPC networking for AWS Lambda functions

How can I troubleshoot connectivity issues when using NAT gateway on my private VPC?

AWS OFFICIAL
AWS OFFICIALAktualisiert vor einem Jahr