How can I troubleshoot API HTTP 504 timeout errors with API Gateway?

4 minute read
0

I receive an HTTP 504 error status code when I invoke my REST API, HTTP API or Websocket API using Amazon API Gateway.

Short description

If an integration request takes longer than your API Gateway REST API maximum integration timeout parameter, API Gateway returns an HTTP 504 status code.

To troubleshoot 504 timeout errors from API Gateway, first identify and verify the source of the error in your Amazon CloudWatch execution logs. Then, use one or more of the following methods to reduce the runtime of your integration requests until they don't timeout.

Resolution

To identify and verify the source of the 504 error in your Amazon CloudWatch logs

1.    For Rest API and Websocket API, set up API Gateway execution logging for the 504 errors. For HTTP API, activate logging to write logs to CloudWatch logs.

2.    Manually try to reproduce the 504 error in the API.

3.    Activate access logging for the API and then use the following parameter variables to diagnose the source of error:

$context.integration.status: The status code returned from an integration. For AWS Lambda proxy integrations, this is the status code that your Lambda function code returns.
$context.integrationStatus: For Lambda proxy integration, this parameter represents the status code returned from Lambda, not from the backend Lambda function.
$context.integrationLatency: The integration latency in ms.

For more information, see $context Variables for data models, authorizers, mapping templates, and CloudWatch access logging.

4.    Use the following CloudWatch Log Insights query to filter the ‘5XX’ status code from access logs:

fields @timestamp, @message, @logStream
| filter status like ‘5’
| sort @timestamp desc
| display @timestamp,httpMethod,resourcePath,status,extendedRequestId,requestId

5.    In the CloudWatch console, view the API Gateway execution logs for the integration receiving the error.

6.    Track the request ID in your CloudWatch logs. If there are timeouts on the integration, you see an "Execution failed due to a timeout" error after the "Endpoint request body after transformations:" line. For more information, see How do I find API Gateway REST API errors in my CloudWatch logs?

7.    Use the following CloudWatch Log Insights query to filter the error and select your API Gateway execution log group:

fields @timestamp, @message
|filter @message like "Execution failed due to a timeout error"
|sort @timestamp desc

8.    To determine the source of the error, verify that the associated integration endpoint was invoked.

9.    Confirm how long the integration took to complete the processing of the request and respond to API Gateway.

10.    If the integration wasn’t invoked, implement API retries on the client. (The error might have resulted from a temporary network failure in the API Gateway service.)

Note: Make sure that your application is idempotent. This avoids data conflicts when you retry the API request.

-or-

If the integration was invoked but still returned a 504 error message, try reducing your integration's runtime. If it's an HTTP API, you can try increasing your maximum integration request timeout parameter.

Note: API Gateway REST API’s default maximum integration timeout is 29 seconds. For HTTP API, the timeout can be configured for up to the maximum value of 30 seconds. The maximum value limits can’t be increased.

To reduce your integration's runtime

  • Make sure that your backend integration includes only the logic needed for API Gateway to send an HTTP response to the client. Consider moving any non-dependent or post-processing logic to another service, such as Lambda.
  • If network latencies are causing the 504 error, implement retry logic on the client-side application.
  • Improve the backend integration performance by following the optimization best practices for your platform.
  • Consider setting up asynchronous invocation of the backend Lambda function.

Related information

Setting up REST API integrations

Setting up a WebSocket API integration request in API Gateway

Configuring integrations for HTTP APIs

Amazon API Gateway quotas and important notes

2 Comments
  1. To determine the source of the error, verify that the associated integration endpoint was invoked.

How do i enable the integration endpoint connect traces? weather the endpoint was connection time out ( waiting for a connection or trying to send the request payload ( read timeout on the other end) or the response read timeout?

for ex: between the below 2 log statements - request was being sent at 09th sec and 504 timeout at 38the sec after 29 secs which is valid since the backend did not response with in 29 secs. Question s how do i know that on the http call - weather backend api provided the connection and the request is sent & just waitng for response to be read? or the waiting for connection itself ? where do i see connect timeout and read timeouts?

42:09.2 (f4267199-574c-4d30-b6b9-25e26d687255) Sending request to https://mulesoft1.directv.com/dtvcpm-payments-xapi-v1/payment d5ec1e06b0a5c67b0e01632ae9602bad 787460153767:API-Gateway-Execution-Logs_65g1naepdi/prod

42:38.2 (f4267199-574c-4d30-b6b9-25e26d687255) Execution failed due to a timeout error d5ec1e06b0a5c67b0e01632ae9602bad 787460153767:API-Gateway-Execution-Logs_65g1naepdi/prod

Ila
replied 6 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 6 months ago