Cloudwatch agent configuration for scraping prometheus metrics from microservices

Question

Hello,

I am attempting to scrape metrics from my microservices on my EKS cluster and
post those to AWS Cloudwatch. I deployed AWS Observability add-on for EKS which
routes my logs and _infrastructure_ metrics to AWS Cloudwatch but not my
_application_ metrics. I noticed that the cloudwatch-agent configmap in the
amazon-cloudwatch namespace which looks like the following doesn't include
section for metrics.

```
{
  "agent": {
    "region": "us-east-2"
  },
  "logs": {
    "metrics_collected": {
      "application_signals": {
        "hosted_in": "my-test-eks-cluster"
      }
    }
  },
  "traces": {
    "traces_collected": {
      "application_signals": {}
    }
  }
}
```

I followed the guides at:
1. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus.html
2. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-configure-ECS.html

and as my microservice runs as a Daemonset, I added the following stanza to the
cloudwatch-agent AmazonCloudWatchAgent resource to allow cloudwatch-agent to
scrape metrics from my services.

```
      "metrics": {
        "namespace": "Prometheus Metrics",
        "metrics_collected": {
          "prometheus": {
            "prometheus_endpoint": "http://localhost:9191/metrics",
            "metric_declaration": [
              {
                "source_labels": ["instance", "job"],
                "label_matcher": "^.*$",
                "dimensions": [["InstanceId", "job"]],
                "name": "cpu_usage",
                "unit": "None"
              },
	      ...
```

However the Amazon Cloudwatch Agent controller didn't like the config and spewed
out several errors including:

```
2025/04/03 10:49:52 Reading json config file path: /etc/cwagentconfig/cwagentconfig.json ...
2025/04/03 10:49:52 E! Invalid Json input schema.
2025/04/03 10:49:52 E! Invalid Json input schema.
2025/04/03 10:49:52 Under path : /metrics/metrics_collected/prometheus | Error : prometheus_config_path is required
2025/04/03 10:49:52 Under path : /metrics/metrics_collected/prometheus | Error : Additional property metric_declaration is not allowed
2025/04/03 10:49:52 Under path : /metrics/metrics_collected/prometheus | Error : Additional property prometheus_endpoint is not allowed
2025/04/03 10:49:52 Configuration validation first phase failed. Agent version: 1.0. Verify the JSON input is only using features supported by this version.
2025/04/03 10:49:53 I! Return exit error: exit code=1
2025/04/03 10:49:53 E! Cannot translate JSON, ERROR is exit status 1
```

Does anybody seem what am I doing wrong? I would appreciate a pointer to what I might be missing.

Alternatively, I was wondering if it makes more sense to setup a prometheus
server to scrape metrics from my services and and a Amazon Managed Prometheus
(AMP) to collect prometheus metrics from my services.

Thank you,

re:Post Agent · Accepted Answer

Based on the information provided and the errors you're encountering, there are a few key points to address regarding the CloudWatch agent configuration for scraping Prometheus metrics from your microservices on an EKS cluster:

1. The configuration you're trying to use seems to be mixing elements from different setups. The CloudWatch agent configuration for Prometheus metrics in EKS clusters is different from the one used in ECS.

2. For EKS clusters, the CloudWatch agent configuration file should have a 'prometheus' section under 'metrics_collected'. This section should include:

- cluster_name (optional): Specifies the cluster name to be added as a label in the log event.
   - log_group_name (optional): Specifies the log group name for the scraped Prometheus metrics.
   - prometheus_config_path: Specifies the Prometheus scrape configuration file path.

3. The error messages you're seeing indicate that the 'prometheus_config_path' is required but missing in your configuration. Also, 'metric_declaration' and 'prometheus_endpoint' are not allowed in this context.

4. For EKS, you typically define your Prometheus scrape targets in a separate Prometheus configuration file, not directly in the CloudWatch agent config.

To resolve this, you should:

1. Separate your Prometheus scrape configuration from the CloudWatch agent configuration.
2. In the CloudWatch agent config, specify the path to your Prometheus config file using the 'prometheus_config_path' field.
3. Create a separate Prometheus scrape configuration file with your target endpoints and other Prometheus-specific settings.

Regarding your alternative approach of setting up a Prometheus server and using Amazon Managed Prometheus (AMP), this is indeed a valid option. It can provide more flexibility in terms of metric collection and querying. However, the choice between direct scraping with the CloudWatch agent and using AMP depends on your specific needs, such as the scale of your metrics, query requirements, and integration with other AWS services.

If you decide to stick with the CloudWatch agent approach, make sure to review the documentation specifically for EKS Prometheus metric collection, as it differs from the ECS setup.  
**Sources**  
[Scraping additional Prometheus sources and importing those metrics - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-configure.html)  
[Set up and configure Prometheus metrics collection on Amazon EKS and Kubernetes clusters - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-install-EKS.html)

Cloudwatch agent configuration for scraping prometheus metrics from microservices

Add your answer

Relevant content