By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

NLB creates with private IPs in 3 AZs, but 2 of them disappear during its lifetime

0

The following is the timeline of the events:

  1. Private NLB is created across 4 AZs, 3 of them populated with EC2 instances which are part of the NLB's target groups.
  2. Over time, the pool of instances shrinks down to occupy a single AZ.
  3. Sometime later, we notice that the NLB's DNS record now resolves to a single A record pointing to an IP within the same AZ as the one containing the EC2 instances. This seems unusual, because NLB's private IPs are supposed to remain static after the LBs creation as I've confirmed in this re:Post question
  4. New EC2 instances are brought up in other AZs and are added to the NLBs target groups. The LB does not adjust to this by adding new IPs in these AZs; it continues to resolve to a single A record.

So my questions related to the timeline above are:

  1. Is the disappearance of previously attached IPs an expected part of the NLBs lifecycle? How does this square with the statement about NLB's private IPs being static?
  2. Should I expect this NLB to remediate and add in private IPs across newly occupied AZs?
  3. If all instances in the same AZ as the NLB's last remaining IP are taken down, will this NLB continue to sending traffic to the other instances if I keep cross-zone load balancing turned off?
Topics
ComputeNetworking & Content Delivery
Tags
Amazon EC2Networking & Content DeliveryElastic Load Balancing
Language
English
stanbsky-cb
asked a year ago604 views
1 Answer
3
Accepted Answer

I think you'll find that the IP addresses are still in the VPC and allocated to NLB. What you're seeing is NLB remove the IP addresses from the DNS responses because there are no healthy targets for those NLB nodes.

Once new instances replace the unhealthy targets the DNS records will be updated. Note it may take some time for the health checks to pass.

profile pictureAWS
EXPERT
Brettski-AWS
answered a year ago
profile pictureAWS
EXPERT
Tushar_J
reviewed a year ago
  • Thaddeus_W EXPERT
    a year ago

    Definitely agree with Brett here, but I would add that this is the desired behavior because you have cross-zone load balancing turned off. You wouldn't want an NLB in an AZ without a healthy target to get traffic it can't route. If cross zone load balancing was desirable and enabled the IPs would not be removed from DNS as they would still have healthy targets in the one AZ. I would also call out that if there were no NLBs with healthy targets there is a fail safe back to responding to DNS with all the NLB IP addresses.

    I think the answers to question 2/3 are implied, but the NLB DNS entry will start responding with the IPs of the addtional AZs of NLB once health checks pass and for 3 assuming cross zone load balancing is disabled the NLBs will not route requests across zones, but DNS will only respond with the IPs of AZs NLB that have healthy targets, so you shouldn't end-up in this situation if you are using the provided DNS.

  • stanbsky-cb
    a year ago

    Thank you, you're right. I managed to find the NLB associated IPs under EC2 -> Network Interfaces. Regarding the DNS record updates, I've had a healthy test instance from a different AZ attached to the NLB for over a week now. The NLB's DNS still reports that single A record as mentioned.

  • Brettski-AWS EXPERT
    a year ago

    Please raise a support case about that - it's important to find out why that is happening.

  • stanbsky-cb
    a year ago

    Thank you Brettski, you've been very helpful. I'll follow your suggestion and raise this with support.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content