App Runner randomly drops requests from API Gateway
I have a service deployed in App Runner that serves as HTTP proxy integration behind API Gateway. This setup worked perfectly fine for months, but starting around May 25th, App Runner started to drop some requests that are coming from the API Gateway. Error rates spiked and integration latency went up to around 5 seconds, up from 1 second, due to the dropped requests I assume. All the deployed binaries are the exact same and the latency on the service that sits behind App Runner stayed flat.
So it looks like something must have changed in App Runner. It almost behaves like a rate limit or some firewall rule where it blocks requests after a certain amount of requests. But hard to troubleshoot since there is limited visibility into the App Runner service. Is there a way to allow those requests in case there are some blocking rules in place?
This seems reproducible. I tried redeploying the service, but still get the same behavior. Any ideas what else I can try? I would love to keep using the service.
Is it possible to provide additional information on why you are failing?
- API Gateway execution log
- App Runner application log
Here are some more infos:
The API gateway returns a 504 gateway timeout response. When the App Runner enters this state where it blocks requests, it also does not return responses when I send a HTTP request to the App Runner endpoint directly. The application logs don't show anything in this state since those requests don't seem to reach the container. I also don't see a scaling event in the App Runner service.
I have now deployed the exact same docker image in a Fargate container behind an Application Load Balancer and just changed the integration URL in the API Gateway. The problem is now completely gone.
DNS resolution failure from App Runnerasked 6 months ago
Public APIGW Rest Api proxy to private Fargate App in VPC via VPC Linkasked 6 months ago
Api Gateway with Lambda Integration Failureasked 6 months ago
App Runner randomly drops requests from API Gatewayasked a month ago
App Runner Deployment Race Condition When Using CloudFormation - No Outbound Connectivityasked 6 months ago
How to assign same custom domain to my amplify app and the API gateway that it usesasked a month ago
App Runner Scaling for background serviceasked 7 months ago
Protect HTTP Api Gateway with WAFasked 3 months ago
Binary uploads to API Gateway Proxy with Lambda IntegrationAccepted Answerasked 5 years ago
API Gateway integration random status 500asked 3 years ago