Can I Use NVMe Reservation on a Multi-Attach Enabled Volume?

Question

Hi,

I want to use Amazon EBS Multi-Attach to share data between multiple EC2 instances.

In the [UserGuide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volumes-multi.html), it says
> Multi-Attach enabled volumes do not support I/O fencing. I/O fencing protocols control write access in a shared storage environment to maintain data consistency. Your applications must provide write ordering for the attached instances to maintain data consistency.

I've googled *"I/O fencing"* and found that NVMe Reservation is a good way (compared to "power fencing", which means powering off the error node) to implement I/O fencing.
However, I failed to use NVMe Reservation on a Multi-Attach Enabled Volume.

The details are as follows.

- EC2: r5b.large, ubuntu-22.04
- EBS: io2, 100GiB, 6000 IOPS, enabled Multi-Attach

I run following command in ec2 instance:
```
> sudo nvme list
```
and get
```
Node                  SN                   Model                                    Namespace Usage                      Format           FW Rev
--------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          volxxxxxxxxxxxxxxxx Amazon Elastic Block Store               1           8.59  GB /   8.59  GB    512   B +  0 B   2.0
/dev/nvme1n1          volxxxxxxxxxxxxxxxx Amazon Elastic Block Store               1         107.37  GB / 107.37  GB    512   B +  0 B   2.0
```
then run 
```
> sudo nvme resv-register -n 1 --crkey=0x0 --nrkey=0xa1 --rrega=0 /dev/nvme1n1
```
and get
```
NVMe status: INVALID_OPCODE: The associated command opcode field is not valid(0x2001)
```
I googled this error message and find that it may be caused by the driver not supporting. To confirm that, I run
```
> sudo nvme amzn id-ctrl /dev/nvme1n1 -H
```
and get
```
...
oncs      : 0
  [8:8] : 0	Copy Not Supported
  [7:7] : 0	Verify Not Supported
  [6:6] : 0	Timestamp Not Supported
  [5:5] : 0	Reservations Not Supported
  [4:4] : 0	Save and Select Not Supported
  [3:3] : 0	Write Zeroes Not Supported
  [2:2] : 0	Data Set Management Not Supported
  [1:1] : 0	Write Uncorrectable Not Supported
  [0:0] : 0	Compare Not Supported
...
```
The fifth bit indicates that "not supporting reservation".

So does **NOT** AWS Multi-Attach Enabled Volume support NVMe Reservation? Or there are other ways to solve this problem？

Answer

__https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-reservations.html__

Multi-Attach enabled io2 volumes support I/O fencing. Support for NVMe reservations is enabled by default for all Multi-Attach enabled io2 volumes created after September 18, 2023. To enable support for NVMe reservations for existing io2 volumes created before September 18, 2023, you must detach all instances from the volume and then reattach the required instances. All attachments made after detaching all of the instances will have NVMe reservations enabled. NVMe reservations is supported with the following operating systems:
```
SUSE Linux Enterprise 12 SP3 and later
RHEL 8.3 and later
Amazon Linux 2 and later
```

Answer

Yes! EBS Multi-Attach volumes [support NVMe reservations](https://aws.amazon.com/about-aws/whats-new/2023/09/amazon-ebs-multi-attach-io2-volumes-nvme-reservations/).

You can learn more about the feature, supported commands, and how to set up Windows clusters in the [EBS user guide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-reservations.html) and [Windows driver guide](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/aws-nvme-drivers.html).

Answer

This should be possible now when using io2 volumes: https://aws.amazon.com/about-aws/whats-new/2023/09/amazon-ebs-multi-attach-io2-volumes-nvme-reservations/

Can I Use NVMe Reservation on a Multi-Attach Enabled Volume?

関連するコンテンツ