Do all EBS volumes have only 2 IO queues?


I am deploying SPDK on AWS using EBS volumes, exposing the disks with NVMe over Fabrics.

On SPDK, one can have a variable number of threads to handle the requests and IO. For performance reasons, though, SPDK does not shares the NVMe IO queues between threads, so each thread must have its own queue. This is usually not a problem since often devices have 8+ IO queues.

EBS volumes, however, seem to have only 2 IO queues. This was verified by looking at /sys/class/nvme/nvmeX/queue_count, which is 3 (1 admin queue + 2 IO queues).

This limits SPDK to 2 threads, which is not enough to achieve full performance. For example, using a im4gn.16xlarge instance with 4x local SSDs + 5x EBS volumes, I can get 9.0GB/s write bandwidth when writing directly to the disks (measured with fio). When creating the fabrics devices with the kernel, I can also get 9GB/s (the kernel shares queues between threads). However, when using SPDK, I get 5.3GB/s with one thread and 6.6GB/s with two threads. So clearly more threads (queues) would help.

I tested both gp3 and io2 volume types (including the io2 Block Express types) and they all had 2 IO queues.

So my question is if there is any way to create EBS volume types that have more IO queues.

asked 7 months ago46 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions