CloudWatch ECS ContainerInsights task-level observablity

Question

Hi community,
Recently with growth of a company where I implement Infrastructure/SRE solutions I started looking for advanced features for observability. And first found that CloudWatch implemented new nice features like related resources and clickable dots at metrics graphs, but when it comes to container-level observability, things in CloudWatch become pretty poor.
First, ContrainerInsights console doesn't show you container logs/metrics/traces immediately (like DataDog would do), but provide toggle list that lead you to different tabs of CloudWatch with pre-filled queries, like logs insights or X-ray traces. And this queries are not task-level! (see attached screenshots). You also need to click "run query" to see results.

![Enter image description here](/media/postImages/original/IMhIab4_hdSNKDmP1GmLXGrA)
![Enter image description here](/media/postImages/original/IMizLdpWpuS5-wFJtmpuIa8Q)
![Enter image description here](/media/postImages/original/IMhMTRpRcHT6Ga_tvsYGdaGg)

The last ContainerInsights option is performance logs - which are very difficult to inspect, because they represent values that are much simpler to understand on graphs, as metrics, cpu utilization for example. Logs insights allow you to build visual graph using query results and even add it to dashboard, but of course it's very difficult to automate such stuff and impossible to use on regular basis. 
I understand, that making each performance log field a distinct metric would blow up pricing because of custom metrics dimensions, but I'm looking for comprehensive observability solution inside AWS. Can you suggest something how can use of CLoudwatch be optimized or improved in described cases?

Thanks

re:Post Agent · Answer

You're right that CloudWatch Container Insights has some limitations when it comes to task-level observability for Amazon ECS, especially compared to third-party solutions like DataDog. However, there are some ways to optimize and improve your use of CloudWatch for container monitoring:

1. Custom Queries: While Container Insights doesn't automatically generate task-level metrics, you can use CloudWatch Logs Insights to create custom queries for task-level data. For example, you can use a query like this to get task-level CPU and memory usage:

```
stats avg(CpuUtilized) as CPU, avg(MemoryUtilized) as Mem by TaskId, ContainerName
| sort Mem, CPU desc
```

2. Custom Dashboards: Once you've created useful queries, you can save them and add them to custom CloudWatch dashboards. This allows you to create task-level views that are more immediately accessible than running queries each time.

3. CloudWatch Application Insights: If you're using AWS Serverless Application Model (SAM), you can enable CloudWatch Application Insights. This service uses machine learning to automatically detect and diagnose problems in your applications, including containerized ones, and can create custom dashboards.

4. Container Insights with Enhanced Observability: As of November 2023, there's a new version of Container Insights with enhanced observability for Amazon EKS clusters running on EC2. While this doesn't directly address ECS task-level metrics, it does provide more granular metrics up to the container level, which might be useful if you're using EKS.

5. CloudWatch Agent: For more detailed metrics, you can configure the CloudWatch agent to collect custom metrics from your containers. This gives you more control over what data you're collecting, though it does require more setup.

6. CloudWatch Embedded Metric Format (EMF): You can use EMF to embed custom metrics alongside your log events. This can be a cost-effective way to generate custom metrics without incurring the full cost of custom metrics.

7. Automated Solutions: Look into CloudWatch observability solutions, which offer pre-configured setups for monitoring various workloads. While they might not solve all your needs, they can provide a starting point for more comprehensive monitoring.

While these approaches can help, they do require more manual setup and querying than a fully integrated solution like DataDog. If you need more comprehensive, out-of-the-box container observability within AWS, you might want to consider complementing CloudWatch with AWS X-Ray for distributed tracing, or explore AWS-native third-party integrations available through AWS Marketplace.

Remember, observability is an evolving field, and AWS continues to improve its offerings. Keep an eye on AWS announcements for new features that might address these limitations in the future.  
**Sources**  
[Viewing Container Insights metrics - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-view-metrics.html)  
[Container Insights - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights.html)  
[CloudWatch observability solutions - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Monitoring-Solutions.html)  
[Observability using native Amazon CloudWatch and AWS X-Ray for serverless modern applications | AWS Cloud Operations & Migrations Blog](https://aws.amazon.com/blogs/mt/observability-using-native-amazon-cloudwatch-and-aws-x-ray-for-serverless-modern-applications/)

CloudWatch ECS ContainerInsights task-level observablity

Relevant content