App Runner randomly drops requests from API Gateway
I have a service deployed in App Runner that serves as HTTP proxy integration behind API Gateway. This setup worked perfectly fine for months, but starting around May 25th, App Runner started to drop some requests that are coming from the API Gateway. Error rates spiked and integration latency went up to around 5 seconds, up from 1 second, due to the dropped requests I assume. All the deployed binaries are the exact same and the latency on the service that sits behind App Runner stayed flat.
So it looks like something must have changed in App Runner. It almost behaves like a rate limit or some firewall rule where it blocks requests after a certain amount of requests. But hard to troubleshoot since there is limited visibility into the App Runner service. Is there a way to allow those requests in case there are some blocking rules in place?
This seems reproducible. I tried redeploying the service, but still get the same behavior. Any ideas what else I can try? I would love to keep using the service.
Here are some more infos:
The API gateway returns a 504 gateway timeout response. When the App Runner enters this state where it blocks requests, it also does not return responses when I send a HTTP request to the App Runner endpoint directly. The application logs don't show anything in this state since those requests don't seem to reach the container. I also don't see a scaling event in the App Runner service.
I have now deployed the exact same docker image in a Fargate container behind an Application Load Balancer and just changed the integration URL in the API Gateway. The problem is now completely gone.
Relevant questions
DNS resolution failure from App Runner
asked 6 months agoPublic APIGW Rest Api proxy to private Fargate App in VPC via VPC Link
asked 6 months agoApi Gateway with Lambda Integration Failure
asked 6 months agoApp Runner randomly drops requests from API Gateway
asked a month agoApp Runner Deployment Race Condition When Using CloudFormation - No Outbound Connectivity
asked 6 months agoHow to assign same custom domain to my amplify app and the API gateway that it uses
asked a month agoApp Runner Scaling for background service
asked 7 months agoProtect HTTP Api Gateway with WAF
asked 3 months agoBinary uploads to API Gateway Proxy with Lambda Integration
Accepted Answerasked 5 years agoAPI Gateway integration random status 500
asked 3 years ago
Hi, @ivo
Is it possible to provide additional information on why you are failing?