Frequent HTTP 400 errors from ElasticBeanstalk with Tomcat 9 and Nginx

Question

Hello,

Since last 2 days we are seeing frequent HTTP 400 errors for our application. The error are sporadic without any specific pattern i.e. a request denied with 400 will succeed if tried again (which is making it very difficult to diagnose).

The error report of 400 doesn't provide any clue because it misses the "Exception Report" element and contains only "Status Report" even though the configuration (tomcat) is updated to provide those details.

Observations:
1. Ones those fail with 400 has "Error from Cloudfront" in X-Cache response header
2. Ones those succeed with 200 has "Miss from Cloudfront" (because we disabled caching and is correct).
3. When request fails, we could see an entry in Nginx logs, meaning the request **was not failed** by Cloudfront itself.
4. When request fails, often we see the request reaching application and processed normally within application's context but response fails (**something might be going on between Tomcat and Nginx**)
5. These errors occur only while serving static resources
6. **Interesting observation - When we try after cleaning the browser it worked properly but after logging into AWS console from the same browser in a separate tab, 400 errors started to appear.**
7. When access directly (without Cloudfront), we don't see any 400 errors

What we have done so far:
1. Increased HttpHeaderSize 
2. Custom error reporting to capture actual error (we tried logging/reporting the request and stack trace causing the issue in the error report but no luck)
3. Application is running on Tomcat9 with recommended platform version, tried rebuilding environment couple of times, nothing worked. (Tomcat or any other upgrade is not an option at this point in time and doesn't seem to be relevant)
4. We've disabled WAF (in Cloudfront), no luck.
5. We logged all headers received, nothing seem to be problematic (in our context).

Not sure why AWS (ElasticBeanstalk) started to behave weird (only recently), any help is much appreciated.

PayOffDeals · Accepted Answer

After ruling out other possible causes and zeroing on cashing issues, isuue got disappeared (not sure if Amazon made any fix internally because it doesn't seem tobe in the realm of application code, nor any direct or indirect extensions provided by tomcat for users)..,

Here few observations:

Request reaches proxy (verified log entry), tomcat (verified log entry), application (verified log entry) and leaves tomcat (verified log entry), proxy and CDN to user.

While request is with tomcat, after leaving the application code, we tried to hook a tracer to capture error but for some reason tomcat skips it, logs error (ignoring the configuration for error reporting and custom interceptor ) and leaves ..,

We suspected caching because there has been a pattern where issue was seen mostly when browser sends caching related headers in the request (any request that does not contain these headers succeeded).

Btw, interceptor gets called everytime except when this error occurs which could also mean it was occurring somewhere between the handover from app code and tomcat, and it manages to generate error report ignoring the configuration.

Frequent HTTP 400 errors from ElasticBeanstalk with Tomcat 9 and Nginx

Add your answer

Relevant content