How to monitor Persistent Volume disk usage in EKS using CloudWatch Observability Add-on?

I've created the sidecar container to monitor the PVC disk usage for Prod EKS environment.

Cloudwatch Agent Configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: cwagent-config
data:
  config.json: |
    {
      "metrics": {
        "metrics_collected": {
          "disk": {
            "measurement": [
              "used_percent"
            ],
            "metrics_collection_interval": 60,
            "resources": [
              "/mnt/pvc"
            ]
          }
        }
      }
    }

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-with-sidecar
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-sidecar
  template:
    metadata:
      labels:
        app: nginx-sidecar
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80
          volumeMounts:
            - name: html
              mountPath: /usr/share/nginx/html
        - name: cloudwatch-agent
          image: amazon/cloudwatch-agent:latest
          volumeMounts:
            - name: html
              mountPath: /mnt/pvc
            - name: cwagentconfig
              mountPath: /etc/cwagentconfig
          command: ["/opt/aws/amazon-cloudwatch-agent/bin/start-amazon-cloudwatch-agent"]
          args:
            - "-config"
            - "/etc/cwagentconfig/config.json"
      volumes:
        - name: html
          persistentVolumeClaim:
            claimName: nginx-pvc
        - name: cwagentconfig
          configMap:
            name: cwagent-config

More details available at https://medium.com/@bibincatchme/how-to-monitor-persistent-volume-disk-usage-in-eks-using-cloudwatch-0933e8ec655d

The CloudWatch Observability Add-on for Amazon EKS does not directly provide metrics for Persistent Volume (PV) disk usage out of the box. However, you can still monitor PV disk usage in your EKS cluster using CloudWatch by following these steps:

1. Install the CloudWatch agent as a sidecar container in your pods that use Persistent Volumes. This allows you to collect custom metrics, including disk usage.

2. Configure the CloudWatch agent to collect disk metrics. You'll need to create a configuration file that includes disk-related metrics such as disk_used_percent or disk_free for the specific mount points where your PVs are attached.

3. Use the CloudWatch agent configuration wizard to generate this configuration, or manually add the required metrics to an existing configuration file.

4. Store the CloudWatch agent configuration as a Kubernetes ConfigMap, which can be read by the CloudWatch agent sidecar container.

5. In your pod specification, add the CloudWatch agent as a sidecar container and mount the ConfigMap containing the agent configuration.

6. Once configured, the CloudWatch agent will start sending disk utilization data for your PVs to CloudWatch.

7. You can then create custom dashboards in CloudWatch to visualize the PV disk usage metrics.

Alternatively, you could set up Prometheus monitoring in your EKS cluster and configure the CloudWatch agent with Prometheus support. This approach allows you to expose custom metrics (including PV disk usage) as Prometheus targets, which can then be discovered, scraped, and ingested into CloudWatch by the agent.

Keep in mind that metrics collected by the CloudWatch agent are billed as custom metrics, so consider the potential cost implications when implementing this solution.

While this approach requires some additional setup compared to the built-in metrics provided by the CloudWatch Observability Add-on, it offers a flexible way to monitor PV disk usage within the AWS ecosystem.
Sources
CloudWatch Agent Aggrerate Disk Metric | AWS re:Post
Metrics for Amazon EKS and Kubernetes - AWS Prescriptive Guidance

Hello,

Using the CloudWatch Observability add-on, the metrics listed in this documentation[1] are supported. Currently, I see no Persistent volume metrics natively being supported by CloudWatch Observability.

Instead you can consider using either AWS managed/Open-source Prometheus/Grafana to achieve this use-case.

Kubernetes exposes the following metrics[2] to monitor PVs which can be queried on Prometheus.

kubelet_volume_stats_available_bytes 
kubelet_volume_stats_capacity_bytes 
kubelet_volume_stats_inodes 
kubelet_volume_stats_inodes_free 
kubelet_volume_stats_inodes_used 
kubelet_volume_stats_used_bytes

[1] https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-reference-performance-entries-EKS.html

[2] https://kubernetes.io/docs/reference/instrumentation/metrics/#:~:text=kubelet_volume_stats_available_bytes