Do all EBS volumes have only 2 IO queues?

0

I am deploying SPDK on AWS using EBS volumes, exposing the disks with NVMe over Fabrics.

On SPDK, one can have a variable number of threads to handle the requests and IO. For performance reasons, though, SPDK does not shares the NVMe IO queues between threads, so each thread must have its own queue. This is usually not a problem since often devices have 8+ IO queues.

EBS volumes, however, seem to have only 2 IO queues. This was verified by looking at /sys/class/nvme/nvmeX/queue_count, which is 3 (1 admin queue + 2 IO queues).

This limits SPDK to 2 threads, which is not enough to achieve full performance. For example, using a im4gn.16xlarge instance with 4x local SSDs + 5x EBS volumes, I can get 9.0GB/s write bandwidth when writing directly to the disks (measured with fio). When creating the fabrics devices with the kernel, I can also get 9GB/s (the kernel shares queues between threads). However, when using SPDK, I get 5.3GB/s with one thread and 6.6GB/s with two threads. So clearly more threads (queues) would help.

I tested both gp3 and io2 volume types (including the io2 Block Express types) and they all had 2 IO queues.

So my question is if there is any way to create EBS volume types that have more IO queues.

demandé il y a 2 ans127 vues
Aucune réponse

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions