CloudWatch Agent not creating CloudWatch metrics for prometheus configuration (for gitlab-runner)

Question

I am trying to publish prometheus metrics (provided by gitlab runner) using CloudWatch agent. For this I created the following prometheus scraping config:

```
global:
  scrape_interval: 1m
  scrape_timeout: 10s
scrape_configs:
  - job_name: gitlabrunner
    sample_limit: 10000
    static_configs:
      - targets:
        - "localhost:9151"
```

And the cw agent config:

```
{
	"agent": {
		"debug": true
	},
	"logs": {
		"metrics_collected": {
			"prometheus": {
				"cluster_name": "gitlabrunner",
				"log_group_name": "my-log-group",
				"prometheus_config_path": "/root/prometheus.yaml",
				"emf_processor": {
					"metric_declaration_dedup":true,
					"metric_namespace":"CWAgent-gitlabrunner",
					"metric_unit": {
						"gitlab_runner_concurrent": "Count"
					},
					"metric_declaration": [{
						"source_labels": ["job"],
						"label_matcher": "^gitlabrunner$",
						"dimensions": [["instance"]],
						"metric_selectors": ["^gitlab_runner_concurrent$"]
					}]
				}
			}
		}
	}
}
```

Now, looking in cloudwatch logs, logs are created. Here one example:

```
{
    "ClusterName": "gitlabrunner",
    "ServiceName": "gitlabrunner",
    "Timestamp": "1695813461654",
    "Version": "0",
    "gitlab_runner_concurrent": 1,
    "gitlab_runner_worker_slots_number": 1,
    "go_goroutines": 14,
    "go_memstats_alloc_bytes": 6426560,
    "go_memstats_buck_hash_sys_bytes": 1470735,
    "go_memstats_gc_sys_bytes": 9730064,
    "go_memstats_heap_alloc_bytes": 6426560,
    "go_memstats_heap_idle_bytes": 3219456,
    "go_memstats_heap_inuse_bytes": 8839168,
    "go_memstats_heap_objects": 25396,
    "go_memstats_heap_released_bytes": 2899968,
    "go_memstats_heap_sys_bytes": 12058624,
    "go_memstats_last_gc_time_seconds": 1695813352.4045906,
    "go_memstats_mcache_inuse_bytes": 2400,
    "go_memstats_mcache_sys_bytes": 15600,
    "go_memstats_mspan_inuse_bytes": 108144,
    "go_memstats_mspan_sys_bytes": 113904,
    "go_memstats_next_gc_bytes": 11654840,
    "go_memstats_other_sys_bytes": 578057,
    "go_memstats_stack_inuse_bytes": 524288,
    "go_memstats_stack_sys_bytes": 524288,
    "go_memstats_sys_bytes": 24491272,
    "go_threads": 8,
    "host": "ip-10-105-156-213.eu-central-1.compute.internal",
    "instance": "localhost:9151",
    "job": "gitlabrunner",
    "process_max_fds": 65535,
    "process_open_fds": 9,
    "process_resident_memory_bytes": 45301760,
    "process_start_time_seconds": 1695739648.98,
    "process_virtual_memory_bytes": 784728064,
    "process_virtual_memory_max_bytes": 18446744073709552000,
    "prom_metric_type": "gauge"
}
```

But all of these entries do not contain a "CloudWatchMetrics" section, and no metrics are created.

Why? What could be wrong?
Thanks!

Accepted Answer

Hello, Nathan. 
It looks like you've configured the CloudWatch agent to collect Prometheus metrics and send them to CloudWatch Logs. However, the CloudWatch agent does not automatically create CloudWatch Metrics from CloudWatch Logs. To achieve this, you need to set up a metric filter in CloudWatch Logs that extracts the relevant data from the logs and then create CloudWatch Metrics based on that filter.

**Here are the steps you can follow:**
* Set Up a Metric Filter in CloudWatch Logs:
* Go to the CloudWatch console.
* In the navigation pane, choose "Log groups."
* Select the log group you specified in your CloudWatch agent configuration (e.g., "my-log-group").
* In the log group, choose "Create Metric Filter."
* Define a filter pattern that matches the log entries you want to convert into CloudWatch Metrics. In your case, it could be something like:

```
{ $.job = "gitlabrunner" && $.gitlab_runner_concurrent }
```
* Click "Test Pattern" to verify that your filter pattern correctly matches log entries.

Best regards, Andrii

Answer

Just a thought / question - are you sure you need both metric_selectors and label_matcher - isn't label_matcher conflicting with the metric selector? You are right that the emf_processor section of your configuration should create a log in CloudWatch metrics embedded format, so if it doesn't work, it is likely that the filters you have defined fail to identify which metric you want to send.

CloudWatch Agent not creating CloudWatch metrics for prometheus configuration (for gitlab-runner)

Relevant content