ALB Health Checks port

Question

I have Fargate service with ALB. 
Worker has 2 ports open. 8080 for requests and 8081 for health checks. 
8080 is limiting connections count to prevent overload. So I created separate port for health checks without any limits.

I setup health check port in my CDK project like this 
```
    ingestFargateServicePublic.targetGroup.configureHealthCheck({
      path: '/status',
      protocol: elb2.Protocol.HTTP,
      interval: cdk.Duration.minutes(3),
      timeout: cdk.Duration.minutes(1),
      port: HEALTH_CHECK_PORT.toString(),
    });
```

Target group has required port in settings

![Enter image description here](/media/postImages/original/IMnpaU3isCRiyfrmlpJhfxGQ)

However, whenever I have extremely high number of connections on 8080 I start to see this in Service log

```
service estr-data-ingest-public port **8080** is unhealthy in target-group EStrea-Inges-QIA5E7IOHXFN  due to (reason Health checks failed).
```

So my question is why ALB is not using proper health check port and mentioning data port in the context of health checks?

Answer

When using AWS Fargate with Application Load Balancer (ALB) for ECS services, the health check settings are typically specified at the target group level. It seems like you've correctly configured the health check port for your target group in the CDK project:

```
ingestFargateServicePublic.targetGroup.configureHealthCheck({
  path: '/status',
  protocol: elb2.Protocol.HTTP,
  interval: cdk.Duration.minutes(3),
  timeout: cdk.Duration.minutes(1),
  port: HEALTH_CHECK_PORT.toString(),
});
```

However, it's important to ensure that your ECS service is registered with the correct target group. If your service is still being associated with the target group on port 8080 instead of the health check port, it can lead to the observed behavior.

Here are a few things to check:

ECS Service Configuration:
Verify that when you create your ECS service using the CDK, you are associating it with the correct target group. Make sure that the service is registered with the target group that corresponds to the health check port.

```
ingestFargateServicePublic.registerLoadBalancerTargets({
  containerName: 'your-container-name',
  containerPort: HEALTH_CHECK_PORT,
  targetGroup: ingestFargateServicePublic.targetGroup,
});
```

Check Container Port Mapping:
Ensure that the container definition in your task definition specifies the correct port mapping for the health check port (8081 in your case). The containerPort should match the health check port you specified in the target group health check configuration.

```
containerDefinitions: [
  {
    name: 'your-container-name',
    image: 'your-container-image',
    portMappings: [
      { containerPort: 8080 }, // Data port
      { containerPort: 8081 }, // Health check port
    ],
  },
],
```

Verify Target Group Settings:
Double-check the target group settings in the AWS Management Console to ensure that the health check port is configured correctly.

Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
In the navigation pane, choose "Target Groups" under "Load Balancing."
Select your target group and check the "Health check settings" to verify the port.
If everything appears to be configured correctly and you are still experiencing issues, consider checking the ECS service events and logs for any relevant information that might help diagnose the problem. Additionally, AWS CloudWatch Logs can be a valuable resource for investigating container and service health.

ALB Health Checks port

Relevant content