Requests disappearing between VPC Flow Logs and ALB Logs

0

I am having an issue where a customer cannot connect to our application servers (EC2 instances), which sit behind an EC2 application load balancer. They can see the request go out from their firewall, with no response whatsoever. It just times out.

Our VPC flow logs show many requests from the source address of this customer, with an "ACCEPT OK" and the target address as one of the load balancer private IPs, which means that these messages should be being routed to the load balancer.

I turned on access logs for this load balancer, and the customer source IP is not showing up anywhere in the logs. At first I suspected the messages could be getting dropped due to desync mitigation, but from what I am reading, drops due to desync mitigation would have an entry in the load balancer logs. There is a network ACL on both the VPC and load balancer, but neither have a rule blocking this IP. We do not have any proxy setup that would affect the source IP in the access logs - I can see other customer source addresses just fine.

What else can I check, and how can these messages disappear like this?

Thank you in advance!

EDIT: The originating client used to have a proxy setup that was removed, so it's possible that the requests are somehow malformed. Is it possible that a malformed request would be silently dropped by the ALB without logging? The same originating client can get responses from other websites (like google.com, aws.amazon.com) without issue.

  • Are your targets showing as healthy?

  • Yes! This was only one specific customer. It did not affect our targets. I will edit the question with what we discovered.

mamadev
asked 9 months ago467 views
3 Answers
0
Accepted Answer

Hi all! I am commenting here with an update.

It turns out that the traffic going through the VPC was initiation traffic only and that the response was never making it back to the client. Since the connection never made the establishing round trip, no logs were present on the ALB, since no data flow was initiated. We opened a support ticket with AWS, as it turned out to be an internal routing issue that has now been resolved.

Thank you for your responses!

mamadev
answered 8 months ago
  • I said exactly that on my answer.. “The return packets are being sent else where.”

    Glad it’s resolved.

0

Have you checked the Security Group? It sounds like the traffic was blocked, hence no log. Were you able to ping the LB? https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-update-security-groups.html

profile pictureAWS
EXPERT
answered 9 months ago
  • I have checked the security group, and the rules are: 0.0.0.0/0 and ::/0 for the port being used (in this case 443). I agree that it sounds like the traffic is blocked, but I can't for the life of me figure out where.

    I'm not sure what you mean by "pinging the LB". We tried to hit it from the customer site, where it just times out (their firewall is open and shows the outbound traffic with no response). From anywhere else, we can get to the load balancer just fine.

    I am going to update my question, because it's possible that the originating request is malformed in some way. The client originating the request used to have a proxy setup that was removed. If that's the case, would the ALB drop it without any indication? I would expect it to at least have a log entry with a failure, but perhaps I am wrong.

0

The first thing that jumps out is network ACL.

Feels like an ACL is blocking it or you’ve a route on your network where the return packets are being sent else where.

Check both inbound and outbound rules on the ACL.

Can you share your acl rules? Check there source IP and make sure you’ve no routes which match there source IP/CIDR range

I would also check the WAF logs if you have them on your ALB

profile picture
EXPERT
answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions