How can I make Prometheus metrics collected by AWS Cloudwatch appear as metrics in Cloudwatch?

0

I have a Java service running in Fargate which exports Prometheus metrics via an /actuator/prometheus endpoint. It is my understanding that the only way to get Prometheus metrics into Cloudwatch is to deploy a Cloudwatch Agent in the Fargate cluster, and have the agent collect the metrics.

After much struggle, I have managed to get the Cloudwatch Agent deployed to the cluster. It also seems to collect metrics exported by the service.

However, the metrics ends up in a log group named /aws/ecs/containerinsights/{name}/prometheus. An example of a single log line can be found below.

How do I get those metrics into the "metrics" section of Cloudwatch? Right now, it seems that the metrics only makes it into a log, which provides no real value. I'd like to be able to browse and graph the metrics. I am unable to find these metrics in the Container Insights or the metrics section.

{
    "ClusterName": "[...]",
    "ECS_PROMETHEUS_NAME": "[...]",
    "ECS_PROMETHEUS_PATH": "/actuator/prometheus",
    "ECS_PROMETHEUS_PORT": "9082",
    "LaunchType": "FARGATE",
    "StartedBy": "ecs-svc/8880037603679832257",
    "TaskClusterName": "[...]",
    "TaskDefinitionFamily": "[...]",
    "TaskGroup": "service:[...]",
    "TaskId": "a57784efbb124bcfa4d3291cc51f4694",
    "TaskRevision": "442",
    "Timestamp": "1697115231324",
    "Version": "0",
    "area": "heap",
    "container_name": "[...]",
    "id": "G1 Eden Space",
    "instance": "10.1.177.38:9082",
    "job": "[...]",
    "jvm_memory_committed_bytes": 170917888,
    "jvm_memory_max_bytes": -1,
    "jvm_memory_used_bytes": 11534336,
    "prom_metric_type": "gauge"
}
已提問 7 個月前檢視次數 693 次
1 個回答
1

Hi, seems like that log event does not contain a special field called CloudWatchMetrics, which is needed to materialize metrics from logs. This format is called EMF (Embedded metric format).

The reason why it's missing could be that your agent configuration does not include "emf_processor" and "metric_declaration" ? Could you check that?

An example configuration can be found in this yaml file: https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-awsvpc.yaml , which is linked from this doc: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-install-ECS.html

AWS
已回答 7 個月前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南