How can I make Prometheus metrics collected by AWS Cloudwatch appear as metrics in Cloudwatch?

0

I have a Java service running in Fargate which exports Prometheus metrics via an /actuator/prometheus endpoint. It is my understanding that the only way to get Prometheus metrics into Cloudwatch is to deploy a Cloudwatch Agent in the Fargate cluster, and have the agent collect the metrics.

After much struggle, I have managed to get the Cloudwatch Agent deployed to the cluster. It also seems to collect metrics exported by the service.

However, the metrics ends up in a log group named /aws/ecs/containerinsights/{name}/prometheus. An example of a single log line can be found below.

How do I get those metrics into the "metrics" section of Cloudwatch? Right now, it seems that the metrics only makes it into a log, which provides no real value. I'd like to be able to browse and graph the metrics. I am unable to find these metrics in the Container Insights or the metrics section.

{
    "ClusterName": "[...]",
    "ECS_PROMETHEUS_NAME": "[...]",
    "ECS_PROMETHEUS_PATH": "/actuator/prometheus",
    "ECS_PROMETHEUS_PORT": "9082",
    "LaunchType": "FARGATE",
    "StartedBy": "ecs-svc/8880037603679832257",
    "TaskClusterName": "[...]",
    "TaskDefinitionFamily": "[...]",
    "TaskGroup": "service:[...]",
    "TaskId": "a57784efbb124bcfa4d3291cc51f4694",
    "TaskRevision": "442",
    "Timestamp": "1697115231324",
    "Version": "0",
    "area": "heap",
    "container_name": "[...]",
    "id": "G1 Eden Space",
    "instance": "10.1.177.38:9082",
    "job": "[...]",
    "jvm_memory_committed_bytes": 170917888,
    "jvm_memory_max_bytes": -1,
    "jvm_memory_used_bytes": 11534336,
    "prom_metric_type": "gauge"
}
질문됨 7달 전692회 조회
1개 답변
1

Hi, seems like that log event does not contain a special field called CloudWatchMetrics, which is needed to materialize metrics from logs. This format is called EMF (Embedded metric format).

The reason why it's missing could be that your agent configuration does not include "emf_processor" and "metric_declaration" ? Could you check that?

An example configuration can be found in this yaml file: https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-awsvpc.yaml , which is linked from this doc: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-install-ECS.html

AWS
답변함 7달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠