API Gateway endpoints going cold?

This behavior you're experiencing with API Gateway endpoints is not expected or standard behavior. API Gateway itself doesn't have a "cold start" issue like Lambda functions do. However, there are a few potential reasons for the symptoms you're describing:

1. Lambda Cold Starts: If your Lambda function hasn't been invoked for a while, it might experience a cold start. However, this typically results in slower response times rather than 500 errors, and it wouldn't persist for several minutes.

2. Database Connections: If your Lambda function is connecting to a database, the connection might time out after periods of inactivity. When the function is invoked after a long period, it might fail to establish a new connection quickly enough.

3. API Gateway Caching: If you have caching enabled on your API Gateway, it's possible that an error response is being cached and served repeatedly. However, this would typically clear after the cache TTL expires.

4. Integration Timeout: There might be an issue with the integration between API Gateway and Lambda. If the Lambda function doesn't respond within the integration timeout period, API Gateway will return a 500 error.

5. Permissions: If the IAM roles or permissions have changed or expired, it could cause API Gateway to fail to invoke the Lambda function.

To address this issue:

1. Check your Lambda function's code for any potential issues with database connections or other external services that might fail after long periods of inactivity.

2. Review your API Gateway settings, particularly the integration timeout and any caching configurations.

3. Verify that the IAM roles and permissions for both API Gateway and Lambda are correct and haven't expired.

4. Implement better error handling in your Lambda function to log detailed error information.

5. Consider implementing a "warm-up" mechanism that periodically invokes your less-used endpoints to keep them active.

6. Enable detailed CloudWatch logging for both API Gateway and Lambda to get more insights into where exactly the error is occurring.

If the issue persists after these checks, you may need to contact AWS support for further assistance, as there might be account-specific configurations or limitations at play.

Sources
AWS REST apigateway throws 502 even though integration timeout is set to 59000 ms | AWS re:Post
API Gateway calls returning HTML, direct calls returning JSON | AWS re:Post

Hello.

Are there any errors in the API Gateway log?
https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html

One error I have experienced in the past was when a Lambda function was using a container and had not been executed for a certain period of time, leaving it in a "Pending" state, resulting in an error the first time it was executed.
If you are using container images for your Lambda functions, you can reduce the occurrence of problems by creating something like EventBridge to run your Lambda functions periodically.
https://docs.aws.amazon.com/lambda/latest/dg/images-create.html#images-lifecycle

After you upload a new or updated container image, Lambda optimizes the image before the function can process invocations. The optimization process can take a few seconds. The function remains in the Pending state until the process completes, when the state transitions to Active. You can't invoke the function until it reaches the Active state.

Are you able to run the Lambda by itself and see how it behaves. How long is your Lambda timeout? Note that, API Gateway has a hard limit on the timeout - I think it's 29s. If the Lambda takes longer to run, the API will fail.

Also, you can set "provisioned concurrency" - https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html - for your Lambda and avoid the cold starts. This is effectively what's happening when you run the Lambda multiple times or invoke the API Gateway endpoint.

Most probably your API invokes a Lambda function that is attached to a VPC or function using SnapStart. When this happens, after 30 days of inactivity for VPC lambdas or 14 days for SnapStart lambdas, we move the function to Inactive state and delete external resources, such as the ENI or the snapshot. When you try to invoke the function again after that, it takes some time for the ENI or snapshot to be created again.

I suggest you setup Health Checks that provide end-to-end testing to regularly invoke rarely used endpoints to keep less-used Lambda functions warm.