SageMaker PIPE Mode vs FSx ?

0

Hi, SageMaker supports training data streaming via PIPE mode, and also reading from FSx distributed file system. Those options seem to provide same value: low latency, high throughput.

  • What are the reasons for using one or the other?
  • Do we have any benchmark of PIPE vs FSx for SageMaker, in terms of costs and speed?
AWS
專家
已提問 5 年前檢視次數 665 次
1 個回答
0
已接受的答案

I can think of the following scenarios

Pipemode cons

** UPDATED**

  1. Data Shuffling - In pipe mode you are working with streaming data and hence you cannot perform data shuffle operations unless you are prepared to shuffle within batches (as in wait to read a batch of records and shuffle within the batch in Pipe mode). Of if your data is distributed across multiples files, then you could use Sagemaker data shuffle to perform file level shuffle

  2. Data readers - There are default data readers for pipemode that come with Tensorflow for formats like csv, tfrecord etc. But if you have custom data formats or using a different deep leaning framework, yYou would have to use custom data readers to deal with the raw bytes and understand the logical end of record. You could also use ml-io to see if any of the built-in pipe mode readers work for your usecase

  3. PIPE mode streams the data for each epoch from S3 and hence will be slower than FSX when you run a few epochs

FSX:

  1. FSX works by lazy loading the s3 file and hence it has a start up delay but gets faster during repeated training.

  2. There is no dependency on the framework and your existing code will work as is..

  3. The only con of using FSX is the additional storage costs, but I would almost prefer FSX to pipe mode in most cases.

AWS
專家
已回答 5 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南