I am trying to send prometheus metrics to aws cloudwatch in eks fargate. I deployed kube-prometheus to eks fargate. I followed instruction in https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup.html
Deployment was successful. I saw logs in /aws/containerinsights/<cluster>/prometheus. But no embedded metrics. So I tried to add prometheus api server metrics (not sure if it will help) by following https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-configure.html
After I added the following to ConfigMap of prometheus-config and deploy, I got "error: error parsing prometheus-eks-fargate.yaml: error converting YAML to JSON: yaml: line 160: did not find expected key":
# Scrape config for API servers
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- default
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: kubernetes;https
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: Namespace
- action: replace
source_labels:
- __meta_kubernetes_service_name
target_label: Service
the configMap before is like:
kind: ConfigMap
metadata:
name: prometheus-config
namespace: amazon-cloudwatch
Thanks in advance,
Julie
I do understand that you had added the shared scrape config to configmap and running in to Parsing error. Since this error points to incorrect YAML formatting of the manifest, I believe the scrape config might have been added with wrong indentation or as the error message suggests there might be missing keys. Although it is hard for us to tell without having the complete YAML manifest that you are trying to apply. Please refer to this manifest(that I grabbed from the same link you were referring to - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-configure.html) -- https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks.yaml
Please ensure that your shared scrape config is placed under the below snippet as shown below:
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config namespace: amazon-cloudwatch data:
prometheus config
prometheus.yaml: | global: scrape_interval: 1m scrape_timeout: 10s scrape_configs: ====== Here comes the scrape Config=== ======== ======== ======== ======== ========
Considering that we need additional information to be able to completely address the issue, I would suggest that you raise a support ticket with AWS Premium Support to get assistance with this issue.