SageMaker PIPE Mode vs FSx ?

0

Hi, SageMaker supports training data streaming via PIPE mode, and also reading from FSx distributed file system. Those options seem to provide same value: low latency, high throughput.

  • What are the reasons for using one or the other?
  • Do we have any benchmark of PIPE vs FSx for SageMaker, in terms of costs and speed?
AWS
エキスパート
質問済み 5年前664ビュー
1回答
0
承認された回答

I can think of the following scenarios

Pipemode cons

** UPDATED**

  1. Data Shuffling - In pipe mode you are working with streaming data and hence you cannot perform data shuffle operations unless you are prepared to shuffle within batches (as in wait to read a batch of records and shuffle within the batch in Pipe mode). Of if your data is distributed across multiples files, then you could use Sagemaker data shuffle to perform file level shuffle

  2. Data readers - There are default data readers for pipemode that come with Tensorflow for formats like csv, tfrecord etc. But if you have custom data formats or using a different deep leaning framework, yYou would have to use custom data readers to deal with the raw bytes and understand the logical end of record. You could also use ml-io to see if any of the built-in pipe mode readers work for your usecase

  3. PIPE mode streams the data for each epoch from S3 and hence will be slower than FSX when you run a few epochs

FSX:

  1. FSX works by lazy loading the s3 file and hence it has a start up delay but gets faster during repeated training.

  2. There is no dependency on the framework and your existing code will work as is..

  3. The only con of using FSX is the additional storage costs, but I would almost prefer FSX to pipe mode in most cases.

AWS
エキスパート
回答済み 5年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ