We're experiencing intermittent 504 errors after ~1 minute from our API running in Fargate with an ALB in front of it. The time and 504 seems to indicate there was an idle timeout, according to this document: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#http-504-issues. The vast majority of requests go through just fine, but something on the order of 1 in 5,000 will error. However, our API logs are indicating that the request was received, processed, and responded to. This is a sample of a 504 from the ALB's logs. "abcdef" where data redacted.
h2 2024-01-26T14:49:30.637331Z app/api-prod/abcdef 108.49.138.101:36790 10.0.0.210:4700 0.000 -1 -1 504 - 229576 604 "POST https://abcdef/participantTest/audio/score HTTP/2.0" "Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-2:abcdef:targetgroup/api-prod/abcdef "Root=1-65b3c63e" "abcdef.com" "session-reused" 0 2024-01-26T14:48:30.597000Z "forward" "-" "-" "10.0.0.210:4700" "-" "-" "-"
The POST requests are usually around 1-20MBs, and so we expect the request to take a little while for clients to upload, but no where near 1 minute, and usually our endpoints indicate the processing time in our API takes on the order of 500ms to complete.
Are there any common causes for something like this, given how randomly it seems to occur? Is there another way to get more information about the traffic between Fargate and the ALB? What are some other debugging steps we could take?
AWS OFFICIALUpdated 2 years ago
