How can I get maximum I/O performance from my EBS volumes hosted on EC2 Nitro-based instances?

Resolution

1.    Verify that your EBS volume isn't hitting IOPS limits. Latency increases if your volume hits its IOPS limit. Increased latency negatively impacts performance. For information on optimizing volume performance, see How do I optimize the performance of my Amazon EBS Provisioned IOPS volumes?

If you're using a GP2 volume check that your volume hasn't exhausted the burst credits.

2.    To benefit from the performance made possible by NVMe storage, you must run one of the following operating systems. Or, make sure that the kernel version supports an I/O scheduler with multi-queue capability. The most commonly used multi-queue I/O schedulers are kyber, mq-deadline, and bfq (budget fair queue).

• Amazon Linux AMI or higher and kernel 4.12 or newer
• CentOS – 7.0 or higher and kernel 3.10 or newer
• Red Hat - 7.0 or higher and kernel 3.10 or newer
• Ubuntu 19.10 with kernel 5.0 or Ubuntu 18.04.03 with kernel 5.0 onwards multi-queue is enabled by default
• Ubuntu – 16.04 or 16.10 multi-queue schedulers aren't kernel compiled and need separate module loading
• SUSE 12 or SUSE 11 with SP3 or higher
• Windows Server 2008 R2, 2012 R2, and 2016 or higher

Note: For operating systems such as Oracle, Linux, Debian, and so on, make sure that you use a kernel version that includes or supports a multi-queue I/O scheduler. The CentOS operating system and its kernel version supports a multi-queue I/O scheduler.

If you're using an operating system older than those listed, you might see a slight decline in the I/O performance. This is because the I/O scheduler on CentOS 6 is a simple scheduler that doesn't support multiple queues. Nitro-based instances have multi-queue processing at the host level. This creates incompatibility between the scheduler at the operating system and host levels and might cause performance degradation.

I/O (read or write) requests submitted to the EBS volume travel through several layers before the volume intercepts them. These layers include user space, the virtual file system, the page cache, the block layer, the scheduler, and the device driver. Tests and benchmarking results using blktrace, blkparse, and btt tools show a slight delay in the I/O scheduler (I2D) layer when using older kernel versions with non multi-queue schedulers on Nitro-based instances.

The I/O scheduler plays an important role in merging, processing, and queuing I/O’s. In Nitro-based systems, EBS volumes are exposed as NVMe block devices. NVME natively supports multiple hardware submission and completion queues. CentOS 7 has the Multi-Queue Block I/O Queueing Mechanism (blk-mq) that allows device drivers to map I/O requests to multiple hardware or software queues. This improves the I/O performance on Nitro-based instances because of the kernel's multi-queuing capabilities. Earlier kernel versions don't have this capability. Because of this, it's a best practice to use a modern operating system with the latest kernel for maximum performance on Nitro-based systems.

I/O scheduler on CentOS 6

$cat /sys/block/xvdf/queue/scheduler noop anticipatory deadline [cfq]
$cat config-2.6.32-754.30.2.el6.x86_64 | grep -i blk_mq

Note: The config file in the CentOS 6 kernel doesn't return the blk_mq as it is using noop scheduler**.**

I/O scheduler on CentOS 7, kernel 3.10 +

$cat /sys/block/nvme1n1/queue/scheduler [none] mq-deadline kyber

Note: Kyber and mq-deadline are multi-queue schedulers present only in higher kernel versions (3.10+ for CentOS 7).

To update the scheduler at the operating system level during the run time of the EC2 instance, use the following command:

#sudo echo 'kyber'> /sys/block/xvdf/queue/scheduler

To change the I/O scheduler permanently, modify the grub configuration and update the elevator parameter in the following format:

Note: The following steps are for CentOS and RHEL.

1.    Run the following command:

#sudo vim /etc/default/grub
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet elevator=kyber"

2.    Run the following command:

#sudo grub2-mkconfig -o /boot/grub2/grub.cfg

The I/O scheduler remains set even if the EC2 instance is rebooted.