Can I Use NVMe Reservation on a Multi-Attach Enabled Volume?



I want to use Amazon EBS Multi-Attach to share data between multiple EC2 instances.

In the UserGuide, it says

Multi-Attach enabled volumes do not support I/O fencing. I/O fencing protocols control write access in a shared storage environment to maintain data consistency. Your applications must provide write ordering for the attached instances to maintain data consistency.

I've googled "I/O fencing" and found that NVMe Reservation is a good way (compared to "power fencing", which means powering off the error node) to implement I/O fencing. However, I failed to use NVMe Reservation on a Multi-Attach Enabled Volume.

The details are as follows.

  • EC2: r5b.large, ubuntu-22.04
  • EBS: io2, 100GiB, 6000 IOPS, enabled Multi-Attach

I run following command in ec2 instance:

> sudo nvme list

and get

Node                  SN                   Model                                    Namespace Usage                      Format           FW Rev
--------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          volxxxxxxxxxxxxxxxx Amazon Elastic Block Store               1           8.59  GB /   8.59  GB    512   B +  0 B   2.0
/dev/nvme1n1          volxxxxxxxxxxxxxxxx Amazon Elastic Block Store               1         107.37  GB / 107.37  GB    512   B +  0 B   2.0

then run

> sudo nvme resv-register -n 1 --crkey=0x0 --nrkey=0xa1 --rrega=0 /dev/nvme1n1

and get

NVMe status: INVALID_OPCODE: The associated command opcode field is not valid(0x2001)

I googled this error message and find that it may be caused by the driver not supporting. To confirm that, I run

> sudo nvme amzn id-ctrl /dev/nvme1n1 -H

and get

oncs      : 0
  [8:8] : 0	Copy Not Supported
  [7:7] : 0	Verify Not Supported
  [6:6] : 0	Timestamp Not Supported
  [5:5] : 0	Reservations Not Supported
  [4:4] : 0	Save and Select Not Supported
  [3:3] : 0	Write Zeroes Not Supported
  [2:2] : 0	Data Set Management Not Supported
  [1:1] : 0	Write Uncorrectable Not Supported
  [0:0] : 0	Compare Not Supported

The fifth bit indicates that "not supporting reservation".

So does NOT AWS Multi-Attach Enabled Volume support NVMe Reservation? Or there are other ways to solve this problem?

asked 11 days ago24 views
1 Answer

Hi Robert,

NVMe reservations are included in the I/O fencing support comment, and are not supported today on EBS Multi-Attach volumes. If you require shared block storage with I/O fencing support, please take a look at FSx for NetApp ONTAP iSCSI volumes, which support fencing.

answered 10 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions