I have an application load balancer with an https listener that forwards all requests to a target group with instances in two availability zones. The target group's health check expects a 200 response. All of the intervals/thresholds/timeouts for the health check are at defaults.
The monitoring graphs show the target group fluctuating between 1 and 2 healthy instances. The logs on the instances show consistent 200 responses at 30 second intervals. However, sometimes they log requests from load balancer IPs in both AZs, and sometimes only from the load balancer IP in their own AZ. Both instances fluctuate between these two states, and each state can last for hours at a time. There seems to be a regular pattern to the fluctuating target health status. See e.g., https://imgur.com/a/oB73TrK
The fluctuations match up with what I see in the instance request logs. I.e., the times when the graph shows a change from 2 healthy targets to fewer than 2, that's the same time that an instance stops logging cross AZ health checks. The instances are not at all resource constrained. They're pretty much idle.
So cross-zone health checks are not logged intermittently, while same-zone health checks are logged consistently. I am having trouble understanding what might be causing this. I'm contemplating setting up notifications for unhealthy instances, but this seems like a lot of work with potentially little reward if it doesn't tell me why the check failed. E.g., if it says something like "connection timed out" or "response timeout" I still wouldn't know why that was happening intermittently.
Any idea what I should look at?
AWS 공식업데이트됨 일 년 전
