Intermittent cross zone health check issue

0

I have an application load balancer with an https listener that forwards all requests to a target group with instances in two availability zones. The target group's health check expects a 200 response. All of the intervals/thresholds/timeouts for the health check are at defaults.

The monitoring graphs show the target group fluctuating between 1 and 2 healthy instances. The logs on the instances show consistent 200 responses at 30 second intervals. However, sometimes they log requests from load balancer IPs in both AZs, and sometimes only from the load balancer IP in their own AZ. Both instances fluctuate between these two states, and each state can last for hours at a time. There seems to be a regular pattern to the fluctuating target health status. See e.g., https://imgur.com/a/oB73TrK

The fluctuations match up with what I see in the instance request logs. I.e., the times when the graph shows a change from 2 healthy targets to fewer than 2, that's the same time that an instance stops logging cross AZ health checks. The instances are not at all resource constrained. They're pretty much idle.

So cross-zone health checks are not logged intermittently, while same-zone health checks are logged consistently. I am having trouble understanding what might be causing this. I'm contemplating setting up notifications for unhealthy instances, but this seems like a lot of work with potentially little reward if it doesn't tell me why the check failed. E.g., if it says something like "connection timed out" or "response timeout" I still wouldn't know why that was happening intermittently.

Any idea what I should look at?

asked 3 years ago399 views
1 Answer
0

I figured it out. The ephemeral port range allowed for NACL ingress traffic to the ELB was not broad enough. So there were periods of time where health checks across AZs were using source ports that caused the return traffic to be blocked.

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions