How do I troubleshoot API Gateway integration issues for Lambda functions?

Resolution

Turn on API logging

Complete the following steps:

1. Open the API Gateway console.

2. In the navigation pane, choose APIs, and then select your API.

3. In the navigation pane, choose Stages, and then select your stage.

4. Under Logs and tracing, choose Edit.

5. Under Logging level, select a level from the dropdown list.

   Note: For full request and response logs, select the Data tracing option with the logging level set to Error and info logs. It's a best practice not to turn on Data tracing for production APIs because data tracing can log sensitive data.

6. Select Detailed metrics.

7. Select Access logging.

8. For Access Log Destination ARN, enter the Amazon Resource Name (ARN) of an Amazon Data Firehose or an Amazon CloudWatch log group.

   Note: Only REST APIs support the Firehose ARN.

9. Enter a log format.

10. Choose Save.

Determine integration types, verify errors, and take next steps to resolve

Complete the following steps:

1. Determine whether API Gateway uses a Lambda proxy integration or a Lambda custom integration. To verify the integration type, check the value of Lambda proxy integration under Integration request.

2. Verify that the errors in API Gateway correspond to the errors in Lambda. Run the following CloudWatch Logs Insights query to find an error status code during a specified time frame:

   parse @message '(*) *' as reqId, message
       | filter message like /Method completed with status: \d\d\d/
       | parse message 'Method completed with status: *' as status
       | filter status != 200
       | sort @timestamp asc
       | limit 50

3. Run the following CloudWatch Logs Insights query to search for Lambda error logs during the same time frame:

   fields @timestamp, @message
       | filter @message like /(?i)(Exception|error|fail)/
       | sort @timestamp desc
       | limit 20

4. Based on the type of error that you identify in your logs, choose one of the following:
   If you receive the following error, then complete the steps in the Resolve concurrency issues section.

   (#####) Lambda invocation failed with status: 429. Lambda request id: ##########
   () Execution failed due to configuration error: Rate Exceeded.
   (#####) Method completed with status: 500

5. If you receive either of the following errors, then complete the steps in the Resolve timeout issues section.
   For a Lambda custom integration:

   < Integration timeout:
   (#####) Method response body after transformations: {"errorMessage":"2019-08-14T02:45:14.133Z ########-####-####-####-############ Task timed out after ##.01 seconds"}
   > Integration timeout:
   (#####) Execution failed due to a timeout error

6. For a Lambda proxy integration:

   < Integration timeout:
   (#####) Endpoint response body before transformations: {"errorMessage":"2019-08-14T02:50:25.865Z ########-####-####-####-############ Task timed out after ##.01 seconds"}
   > Integration timeout:
   (#####) Execution failed due to a timeout error

7. If you receive the following error, then complete the steps in the Resolve function errors section.

   (#####) Execution failed due to configuration error: Malformed Lambda proxy response
   (#####) Method response body after transformations: {"errorMessage": "Syntax error in module 'lambda_function'"}

Resolve concurrency issues

You receive 429 throttling errors or 500 errors when additional requests come in from API Gateway faster than your Lambda function can scale.

To resolve these errors, analyze the following CloudWatch metrics:

• Count is the total number API Gateway API requests during a specified time frame.
• Throttles are the number of Lambda throttled invocation requests. When all function instances process requests and no concurrency is available to scale up, Lambda rejects additional requests with the TooManyRequestsException error. Throttled requests and other invocation errors don't count as invocations or errors.
• ConcurrentExecutions is the number of Lambda function instances that process events. If this number reaches your concurrent executions quota for the AWS Region, then additional invocation requests throttle. Lambda also throttles invocation requests when the number of function instances reaches the reserved concurrency limit that you configured for the function.
  Note: For more information, see API Gateway metrics and Using CloudWatch metrics with Lambda.

If you set a reserve concurrency for your Lambda function, then increase the reserve concurrency value. Or, remove the reverse concurrency value from the Lambda function. The function then draws from the pool of unreserved concurrent executions.

If you don't set the reserve concurrency in the Lambda function, then check the ConcurrentExecutions metric to find out the usage. For more information, see Lambda quotas.

Resolve timeout issues

The default integration timeout limit is 29 seconds for all API Gateway integrations. You can submit a quota request to increase the default integration timeout limit quota to greater than 29 seconds for Regional APIs and private APIs. However, an integration timeout increase might require a reduction in your Region-level throttle quota for your AWS account.

Note: If you increase the integration timeout limit, then make sure that you change the default timeout value of 29 seconds to the new value. For example, change the default timeout value of 29 seconds in the integrations that require the increase. Then, redeploy the API for the new integration timeout limit to take effect.

When you build an API Gateway API with Lambda integration, you might encounter one of the following scenarios:

• The timeout value is less than the integration timeout value.
• The timeout value is greater than the integration timeout value.

If your Lambda function timeout is less than 29 seconds, then check your Lambda logs to investigate this issue. If your Lambda function must run after 29 seconds, then asynchronously invoke the Lambda function.

For Lambda asynchronous invocation custom integration, complete the following steps:

1. Open the API Gateway console.
2. From the navigation pane, choose APIs, and then choose your API.
3. Choose Resources, and then choose your method.
4. Choose Integration Request.
5. Choose Method Request.
6. Expand HTTP Request Headers.
7. Choose Add header.
8. For Name, enter a name for your header. For example: X-Amz-Invocation-Type
   Important: You must map your header from 'Event'. You must use single quotations.

For Lambda proxy integration, use function A and function B. API Gateway first invokes function A synchronously. Then, function A asynchronously invokes function B. Function A can then return a successful response to API Gateway after it asynchronously invokes function B.

If you use a Lambda proxy integration, then you can change the integration to a custom integration. However, to transform the request or response to your specific format, you must configure the mapping templates. For more information, see Set up asynchronous invocation of the backend Lambda function.

Note: Because an asynchronous Lambda function runs in the background, your client can't directly receive data from a Lambda function. You must have an intermediate database to store persistent data.

Resolve function errors

If you receive a function error when you invoke your API, then check your Lambda function for syntax errors. This error also appears if your Lambda function didn't return a valid JSON object that API Gateway expects for proxy integrations.

From the API Gateway execution logs, review the AWS Integration Endpoint RequestID value in the logs:

(#####) AWS Integration Endpoint RequestId : YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY

Then, run the following CloudWatch Logs Insights query to search for Lambda logs during the same specific time frame:

fields @timestamp, @message, @requestId, @logStream
| filter @requestId = 'YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY'
| sort @timestamp asc

To resolve this error, complete the steps in the Turn on logging for your API and stage section.

Override incorrect REST API status code responses

If API Gateway returns an incorrect status code, then create a mapping template to override the incorrect status code to the correct status code. Override status code responses in non-proxy integrations with REST APIs.

Note: This mapping template configuration applies to REST APIs only. For HTTP APIs, see How do I map the response status codes for API Gateway integrations in HTTP APIs?

For example, if API Gateway returns a 200 status code instead of a 4## or 5## from a Lambda function, then complete the following steps:

1. Open the API Gateway console, and in the navigation pane choose APIs.

2. Choose your REST API, and then choose the Integration response tab.

3. In Integration responses settings, choose Edit.

4. Expand Mapping templates and then choose Add mapping template.

5. For Content type, enter application/json.

6. In the mapping template editor, enter the following code:

   #set($inputRoot = $input.path('$'))
   $input.json("$")
   #if($inputRoot.toString().contains("error"))
   #set($context.responseOverride.status = 400)
   #end

7. Choose Save.

The $context.responseOverride.status parameter overrides the status code to 400 instead of the default mapping in the integration response pane.

For more information, see Override your API's request and response parameters and status codes for REST APIs in API Gateway.

Configure your REST API integrations to return the required CORS headers

To return the required CORS headers in its response, configure your backend Lambda function or HTTP proxy server. You must include allowed domains in the Access-Control-Allow-Origin header value as a list.

For proxy integrations, you can't set up an integration response in API Gateway to modify the response parameters that your API's backend returns. In a proxy integration, API Gateway forwards the backend response directly to the client. You must configure your Lambda function or HTTP integration to return the required CORS headers.

For non-proxy integrations, you must manually set up an integration response in API Gateway to return the required CORS headers. Use the API Gateway console to configure CORS. The console automatically adds the required CORS headers to the configured resource.

For more information, see How do I troubleshoot CORS errors from my API Gateway API?

Binary payload Lambda proxy integrations

A binary payload is anything other than a text payload. For example, a binary payload can be a .jpeg file or a .gzip file. This payload type includes generic binary data such as from a .pdf application, .jpeg image, or .zip application.

To handle binary payloads for Lambda proxy integrations, you must base64-encode your function's response and configure the binaryMediaTypes for your API. To handle binary payloads for non-proxy integrations, you must add the media types to the binaryMediaTypes list of the RestApi resource.

For more information, see Binary media types for REST APIs in API Gateway.

Related information

Handle standard Lambda errors in API Gateway

Handle custom Lambda errors in API Gateway