Sagemaker Training Input mode as FastFile still downloading

Question

I'm trying to train a custom model using the `FastFile` input mode, as my input data is quite large (about 27 GB).
As specified in the Sagemaker SDK doc, I've set the `input_mode` parameter of my `Estimator` object to `FastFile`, and created a small pipeline to expose my hyperparameters (I prefer to use the pipeline in Sagemaker Studio).
When I run this pipeline, a training task is created using my custom image and code to train this model.
I usually use the `ml.g4dn.xlarge` instance for training to speed up the process.
Finally, after the initialization stage, the training task attempts to download the data from the "folder" on the S3 bucket (where my 27 GB of data is stored).
But I clearly specified that I wanted to use FastFile mode, I didn't expect to download the data from my bucket.

So the question is: why is the training job still downloading data from the bucket even though I've enabled FastFile mode?

Answer

SageMaker Fast file mode streams the data directly from S3 when you access the file. From an usability perspective you will still access the files as if they are on disc and SageMaker makes sure to stream the file from S3 when accessed. For your use case using File Mode which does the full copy rather than streaming will be better approach as the initial copy is much faster for datasets less than 100 GB. Please refer to the below blog to determine the right option for your training

https://aws.amazon.com/blogs/machine-learning/choose-the-best-data-source-for-your-amazon-sagemaker-training-job/

Sagemaker Training Input mode as FastFile still downloading

Relevant content