How can I make Prometheus metrics collected by AWS Cloudwatch appear as metrics in Cloudwatch?

0

I have a Java service running in Fargate which exports Prometheus metrics via an /actuator/prometheus endpoint. It is my understanding that the only way to get Prometheus metrics into Cloudwatch is to deploy a Cloudwatch Agent in the Fargate cluster, and have the agent collect the metrics.

After much struggle, I have managed to get the Cloudwatch Agent deployed to the cluster. It also seems to collect metrics exported by the service.

However, the metrics ends up in a log group named /aws/ecs/containerinsights/{name}/prometheus. An example of a single log line can be found below.

How do I get those metrics into the "metrics" section of Cloudwatch? Right now, it seems that the metrics only makes it into a log, which provides no real value. I'd like to be able to browse and graph the metrics. I am unable to find these metrics in the Container Insights or the metrics section.

{
    "ClusterName": "[...]",
    "ECS_PROMETHEUS_NAME": "[...]",
    "ECS_PROMETHEUS_PATH": "/actuator/prometheus",
    "ECS_PROMETHEUS_PORT": "9082",
    "LaunchType": "FARGATE",
    "StartedBy": "ecs-svc/8880037603679832257",
    "TaskClusterName": "[...]",
    "TaskDefinitionFamily": "[...]",
    "TaskGroup": "service:[...]",
    "TaskId": "a57784efbb124bcfa4d3291cc51f4694",
    "TaskRevision": "442",
    "Timestamp": "1697115231324",
    "Version": "0",
    "area": "heap",
    "container_name": "[...]",
    "id": "G1 Eden Space",
    "instance": "10.1.177.38:9082",
    "job": "[...]",
    "jvm_memory_committed_bytes": 170917888,
    "jvm_memory_max_bytes": -1,
    "jvm_memory_used_bytes": 11534336,
    "prom_metric_type": "gauge"
}
asked 7 months ago662 views
1 Answer
1

Hi, seems like that log event does not contain a special field called CloudWatchMetrics, which is needed to materialize metrics from logs. This format is called EMF (Embedded metric format).

The reason why it's missing could be that your agent configuration does not include "emf_processor" and "metric_declaration" ? Could you check that?

An example configuration can be found in this yaml file: https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-awsvpc.yaml , which is linked from this doc: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-install-ECS.html

AWS
answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions