Skip to content

ECS Service Connect Traffic Health not being generated

0

I have a few services in ECS that are using Service Discovery AND Service Connect. The services are working fine and can find each other at runtime, however I'm unable to see any "Traffic Health" on ECS Service dashboard related to the mesh network traffic between the services in Service Connect. These services are already using the http appProtocol and my ECS Task IAM Role has the "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy" attached to it with the Assume Role Trust Policies allowing ecs.amazonaws.com and ecs-tasks.amazonaws.com Service Principals already too.

When I look into CloudWatch metrics used by this dashboard, there are none being ingested and I'm wondering if I am missing any IAM Policy rules to be added in order to get this working.

1 Answer
0

Based on the information provided, it seems that you have correctly set up your ECS services with Service Discovery and Service Connect, but you're not seeing the expected Traffic Health metrics in the ECS Service dashboard. This issue could be related to a few factors:

  1. Metric Generation: ECS only sends metrics for resources that have tasks in the RUNNING state. If your services don't have any running tasks, no metrics will be sent to CloudWatch. Ensure that your services have tasks that are actively running.

  2. Metric Delay: There can be a delay in metric reporting or aggregation. CloudWatch metrics sometimes experience delays, so you might need to wait a bit longer for the metrics to appear. The first availability of CloudWatch metrics can take up to 15 minutes after deployment.

  3. IAM Permissions: While you've mentioned that your ECS Task IAM Role has the AmazonECSTaskExecutionRolePolicy attached, you might need to ensure that the role also has permissions to publish CloudWatch metrics. You may need to add the "cloudwatch:PutMetricData" permission to your task role.

  4. Service Connect Configuration: Double-check your Service Connect configuration to ensure it's correctly set up for metric collection. Verify that the services are properly defined and connected in your Service Connect configuration.

  5. CloudWatch Metrics Namespace: Make sure you're looking at the correct namespace in CloudWatch. ECS Service Connect metrics use the "AWS/ECS" namespace.

  6. Metric Dimensions: When checking for metrics, ensure you're using the correct dimensions. For Service Connect, you should look for dimensions like "DiscoveryName, ServiceName, ClusterName" or "TargetDiscoveryName, ServiceName, ClusterName".

If after verifying these points you still don't see the Traffic Health metrics, you may want to:

  1. Review your ECS task definitions and ensure they're up-to-date.
  2. Check your application and ECS service logs for any errors or warnings related to metric publication.
  3. Consider using AWS X-Ray for more detailed tracing of requests across services, which might provide additional insights.

If the issue persists after these checks, it may be worth contacting AWS support for further assistance, as there could be an underlying issue with the Service Connect metric collection in your specific setup.
Sources
Amazon ECS CloudWatch metrics - Amazon Elastic Container Service
Value of TargetResponseTime for AWS ECS Service Connect Service seems incorrect | AWS re:Post
Proactive scaling of Amazon ECS services using Amazon ECS Service Connect Metrics | Containers

answered 2 years ago

  • I've checked all items and they are all in place. Also added "cloudwatch:PutMetricData" to the Task Role but no luck still. I'm not seeing any "AWS/ECS" namespace in cloudwatch metrics though.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.