Unexpected ECS Fargate Task Failures and Recovery – ap-southeast-1

Question

Hi, I'm wondering if anyone else has experienced ECS Fargate tasks failing in healthy check by ALB today from morning until around 2:30pm (AEST) in the aws region **ap-southeast-1**, followed by a recovery without manual intervention?
Also could someone advise Where and how I can track this issue and investigate the root cause?

Thank you for your help!

re:Post Agent · Accepted Answer

I don't see any specific reports in the provided sources about ECS Fargate task health check failures specifically in the ap-southeast-1 region during the timeframe you mentioned. However, there are several potential causes for ECS Fargate task health check failures and ways to investigate them:

For investigating the root cause of your issue:

1. Check CloudTrail logs to see if there were any API calls that might have affected your tasks during that time period.

2. Review CloudWatch logs for your tasks to identify any error messages or unusual patterns leading up to the failures.

3. Use the DescribeTasks API to get more information about the stopped tasks, though this information is only available for about an hour after the task stops.

4. Examine the health check settings for your target groups to ensure they're correctly configured.

5. Verify security group configurations to ensure they allow inbound traffic from the Application Load Balancer to your tasks on the required ports.

6. Check if there were any resource exhaustion issues (CPU, memory, or storage) that could have caused the tasks to become unresponsive.

7. Look for networking issues that might have affected connectivity between your Fargate tasks and other AWS services.

For tracking service issues:

1. Check the AWS Service Health Dashboard for any reported incidents in the ap-southeast-1 region.

2. If you have AWS Support, you can open a case to inquire about any known issues during that time period.

3. AWS re:Post (formerly AWS Forums) might have other users reporting similar issues.

Since your tasks recovered without manual intervention, it's possible there was a temporary issue with the underlying infrastructure or a transient networking problem that resolved itself.  
**Sources**  
[ECS container health checks failing. | AWS re:Post](https://repost.aws/questions/QUs0xsvKkrQw6yCbiXPs8Gkg/ecs-container-health-checks-failing)  
[ALB target group healthcheck failure for ECS service on Fargate with multiple ports | AWS re:Post](https://repost.aws/questions/QUkRX_DK8KTDmB4kgE969mIg/alb-target-group-healthcheck-failure-for-ecs-service-on-fargate-with-multiple-ports)  
[Fargate Task not responding then stuck in stopping status | AWS re:Post](https://repost.aws/questions/QUcaHE42CmSeiFQHElxJTBXw/fargate-task-not-responding-then-stuck-in-stopping-status)

Shahed Kazi (@shajam) · Answer

Check for the health check and find out why it's failing. You can find that out in CloudWatch. 
If nothing appears on CloudWatch, you can also check VPC Flow Logs and search / filter by ECS IP address and check the status code.
Finally, you can also check if the CPU / RAM metrics for your ECS task is too high and therefore not able to serve requests.

Unexpected ECS Fargate Task Failures and Recovery – ap-southeast-1

Add your answer

Relevant content