Why is CentOS Stream 9 AMI slow?

There could be several reasons why you're experiencing slower disk read performance on the CentOS Stream 9 AMI compared to other Linux distributions like AlmaLinux 9.5. While I don't have specific information about that particular AMI, here are some potential factors that could contribute to the performance difference:

1. Kernel version: CentOS Stream 9 might be using an older or differently configured kernel compared to AlmaLinux 9.5. Kernel versions can significantly impact I/O performance, especially when it comes to disk operations.

2. Default I/O scheduler: Different Linux distributions may use different I/O schedulers by default, which can affect disk read performance. CentOS Stream 9 might be using a less optimal scheduler for your specific workload.

3. File system configuration: The default file system settings, such as mount options or the choice of file system (e.g., ext4, XFS), could be different between the two distributions, leading to performance variations.

4. Instance store initialization: If you're using instance store volumes, it's possible that the CentOS Stream 9 AMI hasn't properly initialized these volumes. Some instance store volumes, particularly NVMe SSDs, can suffer from a first-write penalty unless initialized.

5. Default system configurations: CentOS Stream 9 might have different default configurations for various system parameters that affect disk I/O, such as read-ahead settings, I/O queue depths, or caching mechanisms.

6. Kernel modules or drivers: There could be differences in the kernel modules or drivers loaded by default, which might impact storage performance.

To troubleshoot and potentially improve the performance:

1. Ensure you're using the latest available kernel version for CentOS Stream 9.
2. Check and compare the I/O scheduler being used on both systems.
3. Verify the file system type and mount options being used.
4. If using instance store volumes, try initializing them before running your tests.
5. Compare system configurations related to I/O performance between the two distributions.
6. Use more sophisticated benchmarking tools designed for your specific storage type (e.g., fio for NVMe SSDs) to get a more comprehensive performance picture.

Remember that performance can vary based on the specific EC2 instance type, the storage type (EBS vs. instance store), and the particular workload you're running. It's always a good idea to benchmark with your actual workload to get the most relevant performance data.
Sources
Instance store performance | AWS re:Post
Server internet connection is slower than 2G networks | AWS re:Post
Troubleshooting Amazon EFS performance issues - Amazon Elastic File System

By default the IOPS value is only 100 in that CentOS AMI. Changing volume type to gp3 should fix the issue.