How can I make Prometheus metrics collected by AWS Cloudwatch appear as metrics in Cloudwatch?

0

I have a Java service running in Fargate which exports Prometheus metrics via an /actuator/prometheus endpoint. It is my understanding that the only way to get Prometheus metrics into Cloudwatch is to deploy a Cloudwatch Agent in the Fargate cluster, and have the agent collect the metrics.

After much struggle, I have managed to get the Cloudwatch Agent deployed to the cluster. It also seems to collect metrics exported by the service.

However, the metrics ends up in a log group named /aws/ecs/containerinsights/{name}/prometheus. An example of a single log line can be found below.

How do I get those metrics into the "metrics" section of Cloudwatch? Right now, it seems that the metrics only makes it into a log, which provides no real value. I'd like to be able to browse and graph the metrics. I am unable to find these metrics in the Container Insights or the metrics section.

{
    "ClusterName": "[...]",
    "ECS_PROMETHEUS_NAME": "[...]",
    "ECS_PROMETHEUS_PATH": "/actuator/prometheus",
    "ECS_PROMETHEUS_PORT": "9082",
    "LaunchType": "FARGATE",
    "StartedBy": "ecs-svc/8880037603679832257",
    "TaskClusterName": "[...]",
    "TaskDefinitionFamily": "[...]",
    "TaskGroup": "service:[...]",
    "TaskId": "a57784efbb124bcfa4d3291cc51f4694",
    "TaskRevision": "442",
    "Timestamp": "1697115231324",
    "Version": "0",
    "area": "heap",
    "container_name": "[...]",
    "id": "G1 Eden Space",
    "instance": "10.1.177.38:9082",
    "job": "[...]",
    "jvm_memory_committed_bytes": 170917888,
    "jvm_memory_max_bytes": -1,
    "jvm_memory_used_bytes": 11534336,
    "prom_metric_type": "gauge"
}
feita há 7 meses696 visualizações
1 Resposta
1

Hi, seems like that log event does not contain a special field called CloudWatchMetrics, which is needed to materialize metrics from logs. This format is called EMF (Embedded metric format).

The reason why it's missing could be that your agent configuration does not include "emf_processor" and "metric_declaration" ? Could you check that?

An example configuration can be found in this yaml file: https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-awsvpc.yaml , which is linked from this doc: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-install-ECS.html

AWS
respondido há 7 meses

Você não está conectado. Fazer login para postar uma resposta.

Uma boa resposta responde claramente à pergunta, dá feedback construtivo e incentiva o crescimento profissional de quem perguntou.

Diretrizes para responder a perguntas