NLB throwing 502 errors

Question

We have a hub - spoke setup. 
Hub VPC has an ALB1 setup with a listener on 403 port. This routes traffic to Spoke through VPC Peering
Spoke VPC has an NLB listening from Hub ALB1. And that routes traffic to another ALB2 in the same VPC. 
This ALB then routes traffic to an EC2 instance (managed by Auto scaling group)

We receive 502 errors on ALB1. Nothin on ALB2.
We increased the idle timeout to 4000 seconds on both ALBs
Pasting a sample access log

h2 2023-04-20T13:43:05.784432Z app/prod-spoke-eu-west-2-prod/********** 172.70.162.87:16312 10.1.48.38:443 0.001 0.000 -1 502 - 521 594 "GET https://test.com:443/styles.09694db7fc267b15.css HTTP/2.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 Edg/112.0.1722.48" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-west-2:********:targetgroup/https20230220181107110600000015/68a8aed70eef6a43 "Root=1-64414169-7170cd54773bee4675dd2180" "test1.test.com" "arn:aws:acm:eu-west-2:*********:certificate/d38dcad4-ce46-40cf-9978-1048e870a81c" 0 2023-04-20T13:43:05.783000Z "waf,forward" "-" "-" "10.1.48.38:443" "-" "-" "-"

Answer

listener ports ; 443
Since NLB is listening on TCP rule - there are no Access log created.

Also, the issue is intermittent. Out of 100 approx 15 requests gives 502 response.

We had to use NLB - because ALB1 requires a static IP address to establish connection between 2 VPCs. Out of 2 options - either using NLB or using Lambda function to watch the DNS records - we chose to use NLB.

Targets for all 3 load balancers are always in healthy state. There is not a single failure

Answer

Any logs from your NLB?

Any Acls blocking this? What listener ports do you have on your NLB?

Is the ALB showing available on the NLB in the spoke VPC?
Any reason why you are not using alb to alb? Why have an NLB?

Does ALB 2 security groups allow access from NLB?

NLB throwing 502 errors

Relevant content