CloudWatch Agent not creating CloudWatch metrics for prometheus configuration (for gitlab-runner)

0

I am trying to publish prometheus metrics (provided by gitlab runner) using CloudWatch agent. For this I created the following prometheus scraping config:

global:
  scrape_interval: 1m
  scrape_timeout: 10s
scrape_configs:
  - job_name: gitlabrunner
    sample_limit: 10000
    static_configs:
      - targets:
        - "localhost:9151"

And the cw agent config:

{
	"agent": {
		"debug": true
	},
	"logs": {
		"metrics_collected": {
			"prometheus": {
				"cluster_name": "gitlabrunner",
				"log_group_name": "my-log-group",
				"prometheus_config_path": "/root/prometheus.yaml",
				"emf_processor": {
					"metric_declaration_dedup":true,
					"metric_namespace":"CWAgent-gitlabrunner",
					"metric_unit": {
						"gitlab_runner_concurrent": "Count"
					},
					"metric_declaration": [{
						"source_labels": ["job"],
						"label_matcher": "^gitlabrunner$",
						"dimensions": [["instance"]],
						"metric_selectors": ["^gitlab_runner_concurrent$"]
					}]
				}
			}
		}
	}
}

Now, looking in cloudwatch logs, logs are created. Here one example:

{
    "ClusterName": "gitlabrunner",
    "ServiceName": "gitlabrunner",
    "Timestamp": "1695813461654",
    "Version": "0",
    "gitlab_runner_concurrent": 1,
    "gitlab_runner_worker_slots_number": 1,
    "go_goroutines": 14,
    "go_memstats_alloc_bytes": 6426560,
    "go_memstats_buck_hash_sys_bytes": 1470735,
    "go_memstats_gc_sys_bytes": 9730064,
    "go_memstats_heap_alloc_bytes": 6426560,
    "go_memstats_heap_idle_bytes": 3219456,
    "go_memstats_heap_inuse_bytes": 8839168,
    "go_memstats_heap_objects": 25396,
    "go_memstats_heap_released_bytes": 2899968,
    "go_memstats_heap_sys_bytes": 12058624,
    "go_memstats_last_gc_time_seconds": 1695813352.4045906,
    "go_memstats_mcache_inuse_bytes": 2400,
    "go_memstats_mcache_sys_bytes": 15600,
    "go_memstats_mspan_inuse_bytes": 108144,
    "go_memstats_mspan_sys_bytes": 113904,
    "go_memstats_next_gc_bytes": 11654840,
    "go_memstats_other_sys_bytes": 578057,
    "go_memstats_stack_inuse_bytes": 524288,
    "go_memstats_stack_sys_bytes": 524288,
    "go_memstats_sys_bytes": 24491272,
    "go_threads": 8,
    "host": "ip-10-105-156-213.eu-central-1.compute.internal",
    "instance": "localhost:9151",
    "job": "gitlabrunner",
    "process_max_fds": 65535,
    "process_open_fds": 9,
    "process_resident_memory_bytes": 45301760,
    "process_start_time_seconds": 1695739648.98,
    "process_virtual_memory_bytes": 784728064,
    "process_virtual_memory_max_bytes": 18446744073709552000,
    "prom_metric_type": "gauge"
}

But all of these entries do not contain a "CloudWatchMetrics" section, and no metrics are created.

Why? What could be wrong? Thanks!

Nathan
gefragt vor 8 Monaten613 Aufrufe
2 Antworten
0
Akzeptierte Antwort

Hello, Nathan. It looks like you've configured the CloudWatch agent to collect Prometheus metrics and send them to CloudWatch Logs. However, the CloudWatch agent does not automatically create CloudWatch Metrics from CloudWatch Logs. To achieve this, you need to set up a metric filter in CloudWatch Logs that extracts the relevant data from the logs and then create CloudWatch Metrics based on that filter.

Here are the steps you can follow:

  • Set Up a Metric Filter in CloudWatch Logs:
  • Go to the CloudWatch console.
  • In the navigation pane, choose "Log groups."
  • Select the log group you specified in your CloudWatch agent configuration (e.g., "my-log-group").
  • In the log group, choose "Create Metric Filter."
  • Define a filter pattern that matches the log entries you want to convert into CloudWatch Metrics. In your case, it could be something like:
{ $.job = "gitlabrunner" && $.gitlab_runner_concurrent }
  • Click "Test Pattern" to verify that your filter pattern correctly matches log entries.

Best regards, Andrii

profile picture
EXPERTE
beantwortet vor 8 Monaten
profile pictureAWS
EXPERTE
überprüft vor 8 Monaten
  • Hello,

    Thank you for your answer. That actually works, but I still find it strange. From the AWS Documentation I got the impression that metrics should be created automatically, and also that the logs should contain "CloudWatchMetrics" entries. But your solution also works.

0

Just a thought / question - are you sure you need both metric_selectors and label_matcher - isn't label_matcher conflicting with the metric selector? You are right that the emf_processor section of your configuration should create a log in CloudWatch metrics embedded format, so if it doesn't work, it is likely that the filters you have defined fail to identify which metric you want to send.

profile pictureAWS
Jsc
beantwortet vor 8 Monaten
  • You are right, it now also works without the steps from the accepted answer with the config I posted. Mmmh, I don't know what I did different when I posted this, but now the config works.

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen