Load Balancer [NLB] - Listeners - Inconsistent

Question

I'm running my websites in EC2 instance under the NLB [e.g. User (port/443) -> NLB (SSL offload) -> Auto SG -> EC2]

On the LB, when I add a listener [e.g. port/80] then automatically my EC2 instance is not reachable from NLB to EC2.

Even, if I revert back to the added port/80, still it is not reachable of the EC2 instance. however, I can access the EC2 instance with SSH and there are no issues at all.

Even, If I terminate the EC2 instance and my Auto SG will create a new EC2 instance, but still not reachable via the NLTB

So, what I'm doing here to make it reachable via NLB all way to the EC2 instance as follows;

I'll terminate the EC2 instance and wait for a while to auto-create by Auto Scaling Group and then check whether it is reachable or not.

If not reachable, do it again to terminate the process of the EC2 instance.

So is there any workaround for this instead of keeping on terminating the EC2 instance?

Answer

Hi ACP Admin,

From your post I understand that you are running your websites on EC2 instances behind an NLB. And in attempting to connect to this target via the NLB you are seeing inconsistent behavior when using different listeners on the NLB. You specifically mention not being able to connect after adding a listener that is using port 80. You also mention that you are using the NLB as follows "User (port/443) -> NLB (SSL offload) -> Auto SG -> EC2]". However it is not clear from your post whether you are able to connect when using this flow "User (port/443) -> NLB (SSL offload) -> Auto SG -> EC2]". You also inquire whether there is an alternative workaround other than having the instance keep getting terminated by Autoscaling.
For troubleshooting issues on an NLB I recommend reviewing our documentation here[1]. As the exact nature of your issue is not quite clear I also recommend the following steps to assist you in troubleshooting this issue.
1. Check that Targets are passing the NLB healthchecks. If the targets are failing you can check the steps under a "A registered target is not in service" in the referenced documentation.
2. Check the target's Security Group to confirm that the NLB IPs and Client IPs are allowed inbound to the Target Instance.
Note: NLB does not have it's own security group and that it is only secured using the Security Group of the Target instance. Along with the security group you will also need to check the "Client IP Preservation[2]" attribute of the NLB's Target group. If this setting is enabled then the Target instance sees the actual client IP connecting to it and not the NLB IPs. This would mean allowing both the NLB's IPs(for healthchecks) as well as the actual source client IPs in the targets Security Group. If "Client IP Preservation[2]" is disabled then you only need to allow the load balancer IPs inbound to the target instance.
3. Check the Route Tables associated with the NLB subnets. If the NLB is intended to be publicly accessible then the Route Table associated with the NLB subnets requires a default rout 0.0.0.0/0 pointing to an IGW.
4. Check the NACLs associated with the NLB and Target subnets and ensure that the required traffic is being allowed inbound and outbound.
With regards to an alternative to having Autoscaling keep terminating the instance, I recommend removing the Load Balancer from the Autoscaling group[3]. Once it is detached you can launch your target instance behind the NLB and troubleshoot the connectivity issue with the different listeners. Once this issue is resolved you can then look into attaching the NLB to your Autoscaling group once again.

Reference:

[1] Troubleshoot your Network Load Balancer
https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html

[2] Client IP preservation
https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation

[3] Detach a load balancer
https://docs.aws.amazon.com/autoscaling/ec2/userguide/attach-load-balancer-asg.html#as-remove-load-balancer

Load Balancer [NLB] - Listeners - Inconsistent

Relevant content