Which Disk Metrics (nvme0n1p1 vs. nvme1n1p1) Should I Use for CloudWatch Dashboard?

The issue you're experiencing with disk identifiers switching between nvme0n1p1 and nvme1n1p1 is a known problem, particularly with Ubuntu instances on AWS. This switching can occur randomly after rebooting, causing the naming of mapped blocks to change. This behavior can indeed break CloudWatch alarms and make dashboard metrics unreliable if they're based on specific device names.

To address this issue and create more reliable dashboards for disk-related metrics, it's recommended to use the UUID (Universally Unique Identifier) of the volume instead of the device name. Here's why and how to approach this:

1. Use UUID: The UUID of a volume remains constant regardless of how the device is named during boot. This provides a consistent way to identify and monitor your volumes.

2. Update CloudWatch Agent Configuration: Modify your CloudWatch agent configuration to use the UUID or mount point instead of the device name. For example, you can specify the root directory ("/") as the resource to monitor instead of a specific device name.

3. Use the "drop_device" option: In your CloudWatch agent configuration, you can add the "drop_device": true setting. This tells the agent to ignore the device name when collecting metrics, which can help avoid issues caused by device name changes.

Here's an example of how you might update your configuration:

{ "metrics": { "metrics_collected": { "disk": { "measurement": [ "disk_free", "disk_total", "disk_used" ], "metrics_collection_interval": 10, "resources": [ "/" ], "drop_device": true }, "mem": { "measurement": [ "mem_used", "mem_free", "mem_total" ], "metrics_collection_interval": 10 } } } }

By making these changes, your CloudWatch dashboards should become more stable and reliable, regardless of how the disk devices are named after reboots. The metrics will consistently monitor the correct volume, ensuring that your dashboards and alarms function as expected.
Sources
AWs Cloudwatch Alarm Disk Usage Linux EC2 | AWS re:Post