SageMaker PIPE Mode vs FSx ?

I can think of the following scenarios

Pipemode cons

** UPDATED**

1. Data Shuffling - In pipe mode you are working with streaming data and hence you cannot perform data shuffle operations unless you are prepared to shuffle within batches (as in wait to read a batch of records and shuffle within the batch in Pipe mode). Of if your data is distributed across multiples files, then you could use Sagemaker data shuffle to perform file level shuffle

2. Data readers - There are default data readers for pipemode that come with Tensorflow for formats like csv, tfrecord etc. But if you have custom data formats or using a different deep leaning framework, yYou would have to use custom data readers to deal with the raw bytes and understand the logical end of record. You could also use ml-io to see if any of the built-in pipe mode readers work for your usecase

3. PIPE mode streams the data for each epoch from S3 and hence will be slower than FSX when you run a few epochs

FSX:

1. FSX works by lazy loading the s3 file and hence it has a start up delay but gets faster during repeated training.

2. There is no dependency on the framework and your existing code will work as is..

3. The only con of using FSX is the additional storage costs, but I would almost prefer FSX to pipe mode in most cases.