How to push prometheus histogram metrics to cloudwatch

Question

I am using cloudwatch agent to push gitlab-runner metrics to CloudWatch metrics.
The gitlab-runner provides a histogram metric for how long jobs take and also provides this metric for several configured runners, which than looks like this:

```
gitlab_runner_job_duration_seconds_bucket{runner="Tr2JygoJs",system_id="s_91b8bee3abf2",le="30"} 0
gitlab_runner_job_duration_seconds_bucket{runner="Tr2JygoJs",system_id="s_91b8bee3abf2",le="60"} 1
gitlab_runner_job_duration_seconds_bucket{runner="Tr2JygoJs",system_id="s_91b8bee3abf2",le="300"} 9
gitlab_runner_job_duration_seconds_bucket{runner="Tr2JygoJs",system_id="s_91b8bee3abf2",le="600"} 9
gitlab_runner_job_duration_seconds_bucket{runner="Tr2JygoJs",system_id="s_91b8bee3abf2",le="1800"} 9
gitlab_runner_job_duration_seconds_bucket{runner="Tr2JygoJs",system_id="s_91b8bee3abf2",le="3600"} 9
gitlab_runner_job_duration_seconds_bucket{runner="Tr2JygoJs",system_id="s_91b8bee3abf2",le="7200"} 9
gitlab_runner_job_duration_seconds_bucket{runner="Tr2JygoJs",system_id="s_91b8bee3abf2",le="10800"} 9
gitlab_runner_job_duration_seconds_bucket{runner="Tr2JygoJs",system_id="s_91b8bee3abf2",le="18000"} 9
gitlab_runner_job_duration_seconds_bucket{runner="Tr2JygoJs",system_id="s_91b8bee3abf2",le="36000"} 9
gitlab_runner_job_duration_seconds_bucket{runner="Tr2JygoJs",system_id="s_91b8bee3abf2",le="+Inf"} 9
gitlab_runner_job_duration_seconds_bucket{runner="xNY81T-sW",system_id="s_91b8bee3abf2",le="30"} 0
gitlab_runner_job_duration_seconds_bucket{runner="xNY81T-sW",system_id="s_91b8bee3abf2",le="60"} 0
gitlab_runner_job_duration_seconds_bucket{runner="xNY81T-sW",system_id="s_91b8bee3abf2",le="300"} 2
gitlab_runner_job_duration_seconds_bucket{runner="xNY81T-sW",system_id="s_91b8bee3abf2",le="600"} 4
gitlab_runner_job_duration_seconds_bucket{runner="xNY81T-sW",system_id="s_91b8bee3abf2",le="1800"} 5
gitlab_runner_job_duration_seconds_bucket{runner="xNY81T-sW",system_id="s_91b8bee3abf2",le="3600"} 5
gitlab_runner_job_duration_seconds_bucket{runner="xNY81T-sW",system_id="s_91b8bee3abf2",le="7200"} 5
gitlab_runner_job_duration_seconds_bucket{runner="xNY81T-sW",system_id="s_91b8bee3abf2",le="10800"} 5
gitlab_runner_job_duration_seconds_bucket{runner="xNY81T-sW",system_id="s_91b8bee3abf2",le="18000"} 5
gitlab_runner_job_duration_seconds_bucket{runner="xNY81T-sW",system_id="s_91b8bee3abf2",le="36000"} 5
gitlab_runner_job_duration_seconds_bucket{runner="xNY81T-sW",system_id="s_91b8bee3abf2",le="+Inf"} 5
```

Now, I want to push this metric to CW so that I can build a dashboard from this, but separate it for:
a) The runner (so the runner label)
b) The bucket (so le=30,60,....)

How do I have to configure CloudWatch agent to archive this?
The main problem I have is, how can I push the gitlab_runner_job_duration_seconds_bucket for the different values of le as different metrics?

Thanks!

Answer

I think what you need to do here is send the logs to Cloudwatch as they are and then use Cloudwatch itself rather than the Cloudwatch agent to seperate out the data elements you need into Metrics

You might be able to use Metric Filters to achieve the result you are looking for

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringLogData.html

How to push prometheus histogram metrics to cloudwatch

相關內容