App Runner randomly drops requests from API Gateway

0

I have a service deployed in App Runner that serves as HTTP proxy integration behind API Gateway. This setup worked perfectly fine for months, but starting around May 25th, App Runner started to drop some requests that are coming from the API Gateway. Error rates spiked and integration latency went up to around 5 seconds, up from 1 second, due to the dropped requests I assume. All the deployed binaries are the exact same and the latency on the service that sits behind App Runner stayed flat.

So it looks like something must have changed in App Runner. It almost behaves like a rate limit or some firewall rule where it blocks requests after a certain amount of requests. But hard to troubleshoot since there is limited visibility into the App Runner service. Is there a way to allow those requests in case there are some blocking rules in place?

This seems reproducible. I tried redeploying the service, but still get the same behavior. Any ideas what else I can try? I would love to keep using the service.

  • Hi, @ivo

    Is it possible to provide additional information on why you are failing?

    • API Gateway execution log
    • App Runner application log
1 Answer
0

Here are some more infos:

The API gateway returns a 504 gateway timeout response. When the App Runner enters this state where it blocks requests, it also does not return responses when I send a HTTP request to the App Runner endpoint directly. The application logs don't show anything in this state since those requests don't seem to reach the container. I also don't see a scaling event in the App Runner service.

I have now deployed the exact same docker image in a Fargate container behind an Application Load Balancer and just changed the integration URL in the API Gateway. The problem is now completely gone.

ivo
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions