SageMaker PIPE Mode vs FSx ?

0

Hi, SageMaker supports training data streaming via PIPE mode, and also reading from FSx distributed file system. Those options seem to provide same value: low latency, high throughput.

  • What are the reasons for using one or the other?
  • Do we have any benchmark of PIPE vs FSx for SageMaker, in terms of costs and speed?
AWS
전문가
질문됨 5년 전665회 조회
1개 답변
0
수락된 답변

I can think of the following scenarios

Pipemode cons

** UPDATED**

  1. Data Shuffling - In pipe mode you are working with streaming data and hence you cannot perform data shuffle operations unless you are prepared to shuffle within batches (as in wait to read a batch of records and shuffle within the batch in Pipe mode). Of if your data is distributed across multiples files, then you could use Sagemaker data shuffle to perform file level shuffle

  2. Data readers - There are default data readers for pipemode that come with Tensorflow for formats like csv, tfrecord etc. But if you have custom data formats or using a different deep leaning framework, yYou would have to use custom data readers to deal with the raw bytes and understand the logical end of record. You could also use ml-io to see if any of the built-in pipe mode readers work for your usecase

  3. PIPE mode streams the data for each epoch from S3 and hence will be slower than FSX when you run a few epochs

FSX:

  1. FSX works by lazy loading the s3 file and hence it has a start up delay but gets faster during repeated training.

  2. There is no dependency on the framework and your existing code will work as is..

  3. The only con of using FSX is the additional storage costs, but I would almost prefer FSX to pipe mode in most cases.

AWS
전문가
답변함 5년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠