ELB health check frequency

I suggest you open the EC2 console -> Network interfaces, and filter the list by "Description" containing "ELB app/your-alb-name", assuming the load balancer is an ALB. That should give you a quick overview of all the interfaces that the specific load balancer has.

If there are only three ENIs, as expected, then clear the filter and check what "Description" says for some of the IPs that don't belong to the ALB that should be originating the health checks but which are shown in your Wireshark capture.

Are the health checks receiving healthy responses from your application? Have you got more than one target group pointed at the same target, possibly unintentionally causing multiple health checks to target it? Did you account for each load balancer node performing the health checks independently, meaning that if your load balancer has nodes in three AZs, for example, you might see health checks in triplicate.

I have narrowed it down to a single ELB performing the health check every 1 to 2 seconds. The responses are all '200 OK'. Now trying to find out which one it is, but I do not see a valid reason why it would do this.

Based on the information provided, it seems like an issue with the Elastic Load Balancing (ELB) service itself, where it is sending health checks more frequently than the configured interval. Here are a few potential reasons and steps you can take:

1. Check for Multiple Target Groups: Ensure that your target (e.g., EKS service) is not registered with multiple target groups within the same ELB. If it is, you may see health checks from each target group, resulting in a higher frequency.

2. Check for Multiple Load Balancers: If you have multiple load balancers (e.g., an Internet-facing and an Internal ELB) pointing to the same target, you may see health checks from each load balancer, increasing the overall frequency.

3. Check for Load Balancer Node Distribution: As you mentioned, each load balancer node in different Availability Zones (AZs) performs health checks independently. If your load balancer has nodes in multiple AZs, you may see health checks from each node, resulting in a higher frequency.

4. Check for Load Balancer Attributes: Review the load balancer attributes to ensure that no custom settings are overriding the health check interval. You can check the attributes using the AWS Management Console, AWS CLI, or AWS SDKs.

5. Open a Support Case with AWS: If you have verified the above points and still cannot identify the root cause, it's recommended to open a support case with AWS. Provide them with the relevant details, such as the load balancer ARN, target group ARN, and any recent changes made to the configuration. AWS Support may be able to investigate further and provide guidance or escalate the issue if it's a potential bug or issue with the ELB service.

6. Consider Increasing Health Check Interval: As a temporary workaround, you can consider increasing the health check interval for your target group to a higher value (e.g., 30 seconds or 1 minute) to reduce the frequency of health checks. However, this may impact the responsiveness of the load balancer in detecting unhealthy targets.

It's important to note that frequent health checks from the ELB can potentially impact the performance of your application or service, especially if it's under heavy load or has limited resources. Therefore, it's recommended to investigate and resolve the issue to ensure optimal performance and resource utilization.

Sorry, I was a bit too quick with the previous reply but it gets even more weird. The health checks are coming from 37 (!) different src ip addresses and they all have the ELB health checker headers. The IP addresses are within the allocated IP ranges. I have my LB configured to be in 3 AZ's. But it appears there are multiple load balancers per AZ now.

Btw, opening a support case is impossible because it will cost us an insane amount of money for issues that AWS is having. 3% of our monthly bill is an absurd amount of money for a company that never ever asks for support only in cases like this that are inexplicable.