How to access file system in Sagemaker notebook instance from outside of that instance (ie via Python Sagemaker Estimator training call)

Hi Tim, when you create a sagemaker training job using the estimator, the general best practice is to store your data on S3 and the training job will launch instances as requested by the training job configuration. As now we support fast file mode, which allows faster training job start compared to the file mode (which downloads the data from s3 to the training instance). But when you say you used sagemaker notebook instance to train the model, I assume you were not using SageMaker Training jobs but rather running the notebook (.ipynb) on the SageMaker notebook instance. Please note that as SageMaker is a fully managed service, the notebook instance (also training instances, hosting instances etc.) are launched in the service account, so you will not have directly access to those instance. The SageMaker notebook instance use EBS to store data and the EBS volume is mounted to the /home/ec2-user/SageMaker. Please note that the EBS volume used by a SageMaker notebook instance can only be increased but not decrease. If you want to reduce the EBS volume, you need to create a new notebook instance with a smaller volume and move your data from the previous instance via s3. You will not be able to access that EBS volume from outside of the SageMaker notebook instance. The general best practice is to store large dataset on s3 and only use sample data on the SageMaker notebook instance (reduce the storage). Then use that small amount of sample data to test/build your code. Then when you are ready to train on the whole dataset, you can launch a SageMaker training job and use the whole dataset stored on s3. Note that, running the training on the whole dataset on a SageMaker notebook instance will require you to use a big instance with enough computing power and also will not be able to perform distributed training with multiple instances. Comparatively, if you run the training job use SageMaker training instances, it gives you more flexibility of choosing the instance type and allow you to run on multiple instances for distributed training. Lastly, once the SageMaker training job is done, all the resources will be terminated which will save cost compared to continue using the big instance with a SageMaker notebook instance. Hope this has helped answer your question