SageMaker PIPE Mode vs FSx ?

0

Hi, SageMaker supports training data streaming via PIPE mode, and also reading from FSx distributed file system. Those options seem to provide same value: low latency, high throughput.

  • What are the reasons for using one or the other?
  • Do we have any benchmark of PIPE vs FSx for SageMaker, in terms of costs and speed?
AWS
ESPERTO
posta 5 anni fa669 visualizzazioni
1 Risposta
0
Risposta accettata

I can think of the following scenarios

Pipemode cons

** UPDATED**

  1. Data Shuffling - In pipe mode you are working with streaming data and hence you cannot perform data shuffle operations unless you are prepared to shuffle within batches (as in wait to read a batch of records and shuffle within the batch in Pipe mode). Of if your data is distributed across multiples files, then you could use Sagemaker data shuffle to perform file level shuffle

  2. Data readers - There are default data readers for pipemode that come with Tensorflow for formats like csv, tfrecord etc. But if you have custom data formats or using a different deep leaning framework, yYou would have to use custom data readers to deal with the raw bytes and understand the logical end of record. You could also use ml-io to see if any of the built-in pipe mode readers work for your usecase

  3. PIPE mode streams the data for each epoch from S3 and hence will be slower than FSX when you run a few epochs

FSX:

  1. FSX works by lazy loading the s3 file and hence it has a start up delay but gets faster during repeated training.

  2. There is no dependency on the framework and your existing code will work as is..

  3. The only con of using FSX is the additional storage costs, but I would almost prefer FSX to pipe mode in most cases.

AWS
ESPERTO
con risposta 5 anni fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande