EBS io2 Multi-Attach: verifying fencing after a lost Preempt-and-Abort reply

Your pushback on that answer is correct, and I'd go further: the central claim is unsupported and the recommended sequence doesn't close the gap you identified.

On question 1 — no such guarantee is documented

AWS's page on NVMe reservations describes the feature at the level of "industry-standard storage fencing protocols" and lists the supported commands. It doesn't specify completion-ordering semantics, and it doesn't say what Reservation Report's output implies about I/O that was in flight.

The completion guarantee lives in the NVMe specification, not in AWS's documentation, and it's attached to command completion, not to reservation state. Preempt and Abort completes after the abort work is done. That says nothing about what a third party reading Reservation Report can infer, because registration removal and abort drain are separable events and the spec doesn't promise they're atomic to an observer.

So the answer's claim — that seeing B as holder "confirms that the Preempt and Abort operation completed successfully" — conflates state with completion. C is reading state. C never observed B's completion. Nothing in the documentation bridges that.

Your instinct that "seeing a new reservation holder alone doesn't seem sufficient" is right.

On question 2 — it covers only what your command targets

Preempt and Abort aborts commands belonging to the hosts whose registrations that command removes, selected by the preempt key. If B already unregistered A, then A is not a registrant when C runs, C's command doesn't target A, and C's completion carries no statement about A's earlier I/O.

So the proposed sequence — report, register, preempt-and-abort, then mount — does not establish what you need in exactly the case you asked about. It's good practice, and it does guarantee C is the holder going forward, but it doesn't retroactively cover a host that a previous recovery already unregistered.

What that leaves you with

Two things are worth separating, because only one is actually in doubt.

Future writes from A are already stopped. Once A's registration is gone, A's writes fail at the device. There's a useful field report of this: in one implementation on io2 Multi-Attach, a preempted host's next write fails synchronously at write(2) with EBADE, with zero bytes reaching the device. That part is solid and doesn't depend on reconstructing B's outcome.

Only in-flight I/O at the moment of preemption is uncertain, and only its timing. B's command either took effect at the device or didn't — B failing to read the completion queue entry doesn't roll anything back. If Reservation Report shows A unregistered, the device processed the command, and the device will finish the abort work on its own schedule regardless of B's state. What's missing is a documented bound on when.

That reframes the problem usefully: you're not trying to prove an event happened, you're trying to establish that a bounded drain window has elapsed.

Designing around it rather than through it

Since the guarantee you want isn't published, I'd avoid a protocol whose correctness depends on it.

Add an explicit drain interval before C touches the filesystem, derived from the NVMe device and driver command timeouts. Unsatisfying, but it's what cluster stacks actually do, and it's the only thing that addresses in-flight I/O without a completion signal.

Put an epoch in the data. If each writer stamps a monotonically increasing fencing token that the reader validates, a late write from A becomes detectable rather than merely improbable. This makes the storage guarantee a performance concern instead of a correctness one, and it survives arbitrarily many interrupted recoveries — which is the property you said you need.

Consider whether abort is buying you anything. One implementation deliberately uses plain Preempt rather than Preempt and Abort, on the reasoning that aborting outstanding commands is a per-command operation with weaker guarantees than removing the registration, and removing the registration is what stops all future I/O. If you're already adding a drain interval, that argument has some force — the abort variant's extra guarantee is precisely the one you can't observe from C.

STONITH remains the backstop. If A can be powered off or its network detached, the in-flight question disappears. Storage fencing and host fencing solve different halves.

When Host B issues a Reservation Acquire with Preempt and Abort against Host A, the operation atomically revokes Host A's reservation and aborts any outstanding I/O commands from Host A. However, if Host B crashes before receiving the completion, Host C faces the challenge of determining whether the preempt operation completed and whether Host A's I/O has been fully fenced.

The NVMe Reservation Report command can help establish the current reservation state. If Host C issues a Reservation Report and observes that the reservation holder has changed from Host A to Host B, this confirms that the Preempt and Abort operation completed successfully. At that point, Host A's reservation has been revoked and its outstanding I/O commands have been aborted by the storage layer.

However, there are important considerations for your recovery sequence:

First, Multi-Attach enabled io2 volumes support I/O fencing through NVMe reservations, which control write access in a shared storage environment to maintain data consistency. The fencing mechanism ensures that once a host's reservation is preempted, that host cannot issue further successful writes to the volume.

Second, your application must provide write ordering for the attached instances to maintain data consistency. The storage layer's I/O fencing guarantees that preempted hosts cannot write, but your recovery logic must ensure proper sequencing of operations across failover events.

For Host C's recovery sequence, the safest approach is:

1. Issue a Reservation Report to determine the current reservation holder
2. If the reservation is held by Host B (or unregistered), the Preempt and Abort completed
3. Register Host C using Reservation Register
4. Acquire the reservation using Reservation Acquire with Preempt and Abort to fence any remaining hosts
5. Only then access the filesystem

This sequence ensures that Host C has exclusive write access before touching the data, regardless of what happened to Host B. If Host C also crashes, the next recovery host can follow the same sequence.

Regarding the generation counter: While the NVMe specification includes generation counters in reservation reports, the AWS documentation does not provide specific guarantees about generation counter behavior across device lifecycle events, resets, or Multi-Attach scenarios. Relying solely on generation counter increments to infer completion of specific operations may not be safe without explicit documentation of those semantics.

The most reliable approach is to always acquire the reservation with Preempt and Abort before accessing the volume during recovery. This ensures that any previous holder is fenced, regardless of the state of prior operations. This pattern remains safe even across multiple failover events.

For production deployments of this pattern, I recommend testing your specific failover scenarios thoroughly and considering consultation through the AWS Support Center if you need clarification on specific NVMe reservation semantics for your use case.
Sources
Use NVMe reservations with Multi-Attach enabled Amazon EBS volumes - Amazon EBS
Attach an EBS volume to multiple EC2 instances using Multi-Attach - Amazon EBS