Do all EBS volumes have only 2 IO queues?

0

I am deploying SPDK on AWS using EBS volumes, exposing the disks with NVMe over Fabrics.

On SPDK, one can have a variable number of threads to handle the requests and IO. For performance reasons, though, SPDK does not shares the NVMe IO queues between threads, so each thread must have its own queue. This is usually not a problem since often devices have 8+ IO queues.

EBS volumes, however, seem to have only 2 IO queues. This was verified by looking at /sys/class/nvme/nvmeX/queue_count, which is 3 (1 admin queue + 2 IO queues).

This limits SPDK to 2 threads, which is not enough to achieve full performance. For example, using a im4gn.16xlarge instance with 4x local SSDs + 5x EBS volumes, I can get 9.0GB/s write bandwidth when writing directly to the disks (measured with fio). When creating the fabrics devices with the kernel, I can also get 9GB/s (the kernel shares queues between threads). However, when using SPDK, I get 5.3GB/s with one thread and 6.6GB/s with two threads. So clearly more threads (queues) would help.

I tested both gp3 and io2 volume types (including the io2 Block Express types) and they all had 2 IO queues.

So my question is if there is any way to create EBS volume types that have more IO queues.

質問済み 2年前127ビュー
回答なし

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ