Requests disappearing between VPC Flow Logs and ALB Logs

Question

I am having an issue where a customer cannot connect to our application servers (EC2 instances), which sit behind an EC2 application load balancer. They can see the request go out from their firewall, with no response whatsoever. It just times out.

Our VPC flow logs show many requests from the source address of this customer, with an "ACCEPT OK" and the target address as one of the load balancer private IPs, which means that these messages **should** be being routed to the load balancer.

I turned on access logs for this load balancer, and the customer source IP is not showing up anywhere in the logs. At first I suspected the messages could be getting dropped due to desync mitigation, but from what I am reading, drops due to desync mitigation would have an entry in the load balancer logs. There is a network ACL on both the VPC and load balancer, but neither have a rule blocking this IP. We do not have any proxy setup that would affect the source IP in the access logs - I can see other customer source addresses just fine.

What else can I check, and how can these messages disappear like this?

Thank you in advance!

EDIT: The originating client used to have a proxy setup that was removed, so it's possible that the requests are somehow malformed. Is it possible that a malformed request would be silently dropped by the ALB without logging? The same originating client can get responses from other websites (like google.com, aws.amazon.com) without issue.

Accepted Answer

Hi all! I am commenting here with an update.

It turns out that the traffic going through the VPC was initiation traffic only and that the response was never making it back to the client. Since the connection never made the establishing round trip, no logs were present on the ALB, since no data flow was initiated. We opened a support ticket with AWS, as it turned out to be an internal routing issue that has now been resolved.

Thank you for your responses!

Answer

The first thing that jumps out is network ACL.

Feels like an ACL is blocking it or you’ve a route on your network where the return packets are being sent else where.

Check both inbound and outbound rules on the ACL.

Can you share your acl rules?
Check there source IP and make sure you’ve no routes which match there source IP/CIDR range

I would also check the WAF logs if you have them on your ALB

Answer

Have you checked the Security Group? It sounds like the traffic was blocked, hence no log. Were you able to ping the LB? 
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-update-security-groups.html

Requests disappearing between VPC Flow Logs and ALB Logs

Relevant content