Too many ALB health check for EKS pod

Question

I am running a service in an eks environment, and I have a question because too many ALB health check logs are generated in the logs within the pod.
The health check timeout for alb is 5 seconds and the interval is set to 15 seconds.

However, too many health check requests occur. If this is normal, please tell me why. Or What should I do to ensure that health checks are performed at regular intervals?

```
[2024-05-03 05:24:17.136 +00:00] [INF] [88] Request starting HTTP/2 POST http://AWSALB/AWS.ALB/healthcheck - application/grpc 5                                                                           │
│ [2024-05-03 05:24:17.136 +00:00] [INF] [102] Request starting HTTP/2 POST http://AWSALB/AWS.ALB/healthcheck - application/grpc 5                                                                          │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [102] Executing endpoint 'gRPC - Unimplemented service'                                                                                                            │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [142] Request starting HTTP/2 POST http://AWSALB/AWS.ALB/healthcheck - application/grpc 5                                                                          │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [102] Service 'AWS.ALB' is unimplemented.                                                                                                                          │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [102] Executed endpoint 'gRPC - Unimplemented service'                                                                                                             │
│ [2024-05-03 05:24:17.136 +00:00] [INF] [88] Executing endpoint 'gRPC - Unimplemented service'                                                                                                             │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [142] Executing endpoint 'gRPC - Unimplemented service'                                                                                                            │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [142] Service 'AWS.ALB' is unimplemented.                                                                                                                          │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [142] Executed endpoint 'gRPC - Unimplemented service'                                                                                                             │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [102] Request finished HTTP/2 POST http://AWSALB/AWS.ALB/healthcheck - 200 0 application/grpc 0.5566ms                                                             │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [88] Service 'AWS.ALB' is unimplemented.                                                                                                                           │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [88] Executed endpoint 'gRPC - Unimplemented service'                                                                                                              │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [142] Request finished HTTP/2 POST http://AWSALB/AWS.ALB/healthcheck - 200 0 application/grpc 0.363ms                                                              │
│ [2024-05-03 05:24:17.137 +00:00] [INF] [88] Request finished HTTP/2 POST http://AWSALB/AWS.ALB/healthcheck - 200 0 application/grpc 0.8204ms                                                              │
│ [2024-05-03 05:24:17.153 +00:00] [INF] [129] Request starting HTTP/2 POST http://AWSALB/AWS.ALB/healthcheck - application/grpc 5                                                                          │
│ [2024-05-03 05:24:17.153 +00:00] [INF] [129] Executing endpoint 'gRPC - Unimplemented service'                                                                                                            │
│ [2024-05-03 05:24:17.153 +00:00] [INF] [129] Service 'AWS.ALB' is unimplemented.                                                                                                                          │
│ [2024-05-03 05:24:17.153 +00:00] [INF] [129] Executed endpoint 'gRPC - Unimplemented service'                                                                                                             │
│ [2024-05-03 05:24:17.153 +00:00] [INF] [129] Request finished HTTP/2 POST http://AWSALB/AWS.ALB/healthcheck - 200 0 application/grpc 0.1774ms                                                             │
│ [2024-05-03 05:24:17.154 +00:00] [INF] [102] Request starting HTTP/2 POST http://AWSALB/AWS.ALB/healthcheck - application/grpc 5                                                                          │
│ [2024-05-03 05:24:17.155 +00:00] [INF] [102] Executing endpoint 'gRPC - Unimplemented service'                                                                                                            │
│ [2024-05-03 05:24:17.155 +00:00] [INF] [102] Service 'AWS.ALB' is unimplemented.                                                                                                                          │
│ [2024-05-03 05:24:17.155 +00:00] [INF] [102] Executed endpoint 'gRPC - Unimplemented service'                                                                                                             │
│ [2024-05-03 05:24:17.155 +00:00] [INF] [102] Request finished HTTP/2 POST http://AWSALB/AWS.ALB/healthcheck - 200 0 application/grpc 0.1908ms
```

Answer

Hello,

The timing is correct. 5 seconds of Timeout and 15 seconds of delay.
If the health checks are failing you need to check the security group rules for the instances behind the ALB to ensure they allow traffic from the load balancer and verify that the health check path is correct and does not result in a redirect (HTTP 301 response).
Increasing the "Health check grace period" for the target group will also allow more time for your services to start up before the health checks begin.

If they are instead ok you can think of reducing them by adjusting the logging level for the ALB in your EKS cluster. You can do this by modifying the 
`aws-load-balancer-controller
` Helm chart values or the Kubernetes Ingress annotations.
Use a log aggregation service, such as Amazon CloudWatch Logs or Elasticsearch, to centralize and manage your logs more effectively.

Docs: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html

Too many ALB health check for EKS pod

Relevant content