Application Loadbalancer is returning 400 Bad Request

Question

Dear all,

I have a pair of HAProxy, where the Host header gets rewritten (Pubic -> Internal) and forwarded to the Application load-balancer as the `backend`, where based on the host-header, either it returns a `fixed-response` or forwarded to VPC PrivateLink (as the member of the associated target-group). For the fixed-response rule, it returns the correct output based on the rewritten host-header (by HAProxy) but when it goes to  the VPC Endpoint, it returns `400 Bad Request`.  Also to mention that the IPs from the VPC Endpoint as the member of the ALB target-group is also coming as `Unhealthy` for obvious reason(??).

I checked whatever I could and finally came to the decision that nothing wrong with what is being sent from HAProxy, otherwise fixed-response wouldn't work in the first place. Which leaves me thinking something to do with `ALB -> TargetGroup -> VPC Endpoint` but I couldn't figure out anything either.

My backend are just a simple Nginx server, running some vHosts and if I directly access any of those, using the internal domain-name, I get 200 but the moment it comes through the ALB, I see 400 in the log of the `default_server`.  Looks like it's not sending any Host header at all to the backend app. verything is pointing to a `malformed header` but I cannot find anything. Can anyone help me to debug this issue pls?

Answer

Hello.

Can I enable access logging for ALB?  
I believe this log records host headers, etc.  
If the host header is not present at this point, it may have been deleted by HAProxy, and if there is a host header, it may have been deleted by ALB.  
You may also be able to see why the request is failing.  
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html

Answer

It started working now but I'm still not very sure what made it not working in the first place. Few things I noticed along the way:
1. AWS doesn't like if the number of used AZs on the EndPoint Service account (e.g. *account-B*) doesn't match with the number of AZs on the VPC PrivateLink account (e.g. *account-A*)
2. The NLB that associated with the the EndPoint Service (on *account-B*) needs to be cross-zone load balancing enabled
3. Also, I read somewhere that at least two registered targets are required (on *account-B* behind the NLB) to make it work but not very sure about it.

My NLB was cross-zone enabled from the very begining, and I made of point #1 and #3 the moment I git 400 error for the first time. So, all three points were true for me. What I did along the way, disabled and re-enabled NLB cross-zone load balancing capability and it seems like started working at somepoint after that. I'm not entirely sure if that was the case then question still remains why it didn't work before as it was enabled from day one. ALB access-log didn't have any useful or obvious information at all.
Did anyone experience  anything similar at all?

-S

Application Loadbalancer is returning 400 Bad Request

Relevant content