Ingest Prometheus metrics to the workspace

The error you're encountering suggests that there's an issue with the persistent volume configuration for your Prometheus pod. This is likely because the default configuration for the Prometheus chart includes a persistent volume, which isn't compatible with EKS Auto Mode or may not be properly set up in your cluster. To resolve this, you can try the following steps:

1. Disable the persistent volume in your Prometheus configuration. You can do this by adding the following to your my_prometheus_values_yaml file:

server:
  persistentVolume:
    enabled: false

2. If you're using EKS Auto Mode, you need to ensure that you're using the correct EBS CSI driver. The provisioner in your StorageClass should be ebs.csi.eks.amazonaws.com instead of ebs.csi.aws.com.

3. If you still want to use persistent storage, you'll need to create a StorageClass that's compatible with your EKS setup. For EKS Auto Mode, this would look something like:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.eks.amazonaws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3

4. After creating the StorageClass, update your my_prometheus_values_yaml file to use this StorageClass:

server:
  persistentVolume:
    storageClass: "ebs-sc"

5. Make sure that the EBS CSI driver is properly installed and configured in your cluster. In EKS Auto Mode, this should be handled automatically, but it's worth verifying.

6. If you're not using EKS Auto Mode, ensure that you have the Amazon EBS CSI Driver add-on installed in your cluster.

After making these changes, try reinstalling the Prometheus chart using Helm. If you're still encountering issues, you may need to check your cluster's overall storage configuration and ensure that you have the necessary IAM permissions set up for EBS volume management.
Sources
Ingest Prometheus metrics to the workspace - Amazon Managed Service for Prometheus
EKS AutoMode node can't bind to static PV - does not contain driver ebs.csi.aws.com | AWS re:Post