ALB target group healthcheck failure for ECS service on Fargate with multiple ports

0

Hi,

We have a containerised service running on ECS Fargate deployed using AWS copilot. The service has been running without issues. Now we added a MCP (multiple context protocol) server to the container on a different port 8081. The service was able to start up but shut down in a few minutes due to the failed healthcheck for the MCP target group. The container exposes 2 ports, one serves the main logic, one for MCP purpose. I was able to get back 200 ok when requested for healthcheck from the browser using either the target group's private ip address and port (8081) or service's host address (see screenshots attached). Any idea why the ALB healthcheck still failed? Enter image description here Enter image description here Enter image description here Enter image description here

Thanks, PH

asked a month ago63 views
3 Answers
0
Accepted Answer

Managed to run the service with mcp server after I manually adjusted the target group healthcheck settings in the console. Couldn't make it work using the AWS Copilot manifest.yml.

answered 21 days ago
0

The issue you're experiencing with the ALB target group health check failing for your ECS service on Fargate with multiple ports could be due to several factors. Here are some potential reasons and steps to troubleshoot:

  1. Security Group Configuration: Ensure that the security group associated with your ECS task allows inbound traffic on both ports (the original port and the new 8081 port) from the Application Load Balancer's security group.

  2. Task Definition: Verify that your task definition correctly includes both port mappings (the original port and 8081) in the container definition.

  3. Service Configuration: Check if your ECS service is configured to use multiple target group attachments. By default, an ECS service can only register with one target group at a time. You may need to update your service definition to include multiple target group attachments, which requires using the AWS CLI or SDK.

  4. Health Check Settings: Review the health check settings for the new target group. Ensure the health check path, protocol, and port are correctly configured. You might need to adjust the health check grace period or use slow start mode if your application takes time to initialize.

  5. Container Health: Verify that your application is actually listening and responding correctly on both ports within the container.

  6. Network Configuration: Check that the task is running in an Availability Zone that is enabled for the load balancer.

  7. CloudWatch Logs: Examine the application logs in CloudWatch for any error messages or issues that might be causing the health checks to fail.

To resolve this, you may need to:

  1. Update your service definition using the AWS CLI or SDK to properly attach both target groups.
  2. Ensure your task definition includes correct port mappings for both ports.
  3. Verify and adjust security group rules to allow traffic on both ports.
  4. Review and adjust health check settings for the new target group.

If the issue persists after checking these points, you may need to engage AWS support for further investigation.
Sources
ECS Task Registers Only on One Port for Both NLB Target Groups | AWS re:Post
Troubleshoot target deregistration for Amazon ECS tasks | AWS re:Post
Why is registered target health check failing | AWS re:Post

profile picture
answered a month ago
0

If you're able to successfully get 200 OK when manually testing from a browser but health check fails only at the LB level, it appears that the Security Group attached to task is not allowing inbound connections from LB.

AWS
answered 21 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions