FSx performance well-below specification

Question

Hi,

I am currently using two separate FSx (other configs are the same if not mentioned):
A: 120TB persistent SSD, 125MB/s/TB speed, 15000MB/s total throughput
B: 12TB persistent SSD, 1000MB/s/TB speed, 12000MB/s total throughput
Based on the numbers, the two systems seem to offer a similar level of throughput, but in reality, A is 10x faster than B in my use case.
Note that I have run `lfs migrate` on the data directory to make sure the files are organized in FSx-optimized format.

My data directory contains a large set of images (each <1MB), which sums up to 307GB in total.
I test data reading speed with 16 concurrent processes, each process sequentially reads 192 images with Python cv2.imread and counts toward an iteration. 
Here are some results:
A: 2.22 iterations/sec; 
B: 0.245 iterations/sec;

Since B runs at 0.245 iterations/sec, the upper-bound data loading speed should be 192MB * 0.245 iterations/sec = 47MB/sec, which is way below the specified 12000MB/sec total throughput, even lower than 1000MB/s/TB single OST speed.
I look into the CloudWatch, and it reports ~9,000,000/minute DataWriteBytes (Sum). This is equivalent to 150MB/sec summing over 10 OSTs, also lower than 1000MB/s/TB, so it is probably not a data locality issue. The data loading throughput is very stable over time, so it is not bounded by the peak throughput as well.

Does anyone have ideas on what are the possible causes of this, and any suggestions on optimizing this FSx with any commands?
Thanks!

Accepted Answer

In Amazon FSx, the terms "block size" and "stripe size" are used interchangeably and refer to the same concept. Stripe size is the amount of data that is written to each disk in the file system before moving on to the next disk. The larger the stripe size, the larger the amount of data that can be read or written in a single I/O operation.

You can change the stripe size of an Amazon FSx file system when you create the file system or by modifying the file system settings. However, it is not possible to change the stripe size once the file system has been created. Therefore, if you want to change the stripe size of an existing file system, you will need to create a new file system with the desired stripe size and migrate your data to the new file system.

Regarding your specific use case of random access of small files, a larger stripe size may not necessarily improve performance. In fact, as you noted, a larger stripe size can result in a higher penalty on block misses. If your application is primarily performing random access of small files, you may want to consider using a smaller stripe size to improve performance. Additionally, you may want to consider using a caching layer, such as Amazon Elastic Cache or Amazon ElastiCache for Redis, to reduce the number of disk reads required for small files.

Answer

Based on the information provided, it appears that the bottleneck in your use case is likely the small file size of the images you are working with. This is because the I/O operations required to read and write small files are typically much slower than those required for larger files, as there is significant overhead associated with each I/O operation.

One way to improve the performance of FSx with small files is to increase the block size used by the file system. Another potential issue could be the number of concurrent processes reading the data. With 16 concurrent processes, there may be contention for resources, leading to decreased performance. You can try reducing the number of concurrent processes and see if that improves performance.

FSx performance well-below specification

Relevant content