How do I install Python packages in a conda environment on a SageMaker notebook instance?

Resolution

Install Python packages to a specific conda environment

If you're importing the Python package to your running notebook, then you might see a ModuleNotFoundError message. This occurs when you use pip or conda to install Python libraries on the terminal and you don't specify the correct conda environment.

To install the Python packages in the correct conda environment, first activate the environment before you run pip install or conda install from the terminal.

Run the following command to activate the environment:

sh-4.2$ source activate python3(python3) sh-4.2$ pip install theano
(python3) sh-4.2$ source deactivate
(JupyterSystemEnv) sh-4.2$

If you run the command in a notebook cell, add an exclamation point (!) to the beginning of the command. The exclamation point runs the command as a shell to make sure that the package is installed in the current Jupyter kernel.

Run the following conda install command:

import sys
!conda install -y --prefix {sys.prefix} theano

Note: If you run conda install in a notebook cell, then you can't enter an interactive response. To use conda to install packages in a notebook cell, you must explicitly pass -y. Otherwise, the command doesn't respond and waits for user confirmation.

To use pip install, run the following command:

import sys
!{sys.executable} -m pip install theano

If pip doesn't install some of the package's dependencies, then use conda to install the packages instead. Before it installs the packages, conda verifies all required components. For more information, see Understanding conda and pip on the Anaconda website.

Verify prebuilt conda environment

To verify the prebuilt conda environment, run either of the following commands in the notebook instance terminal:

$ conda env list

$ conda info --envs

Example output:

# conda environments:#
base                     /home/ec2-user/anaconda3
JupyterSystemEnv      *  /home/ec2-user/anaconda3/envs/JupyterSystemEnv
R                        /home/ec2-user/anaconda3/envs/R
amazonei_mxnet_p27       /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27
amazonei_mxnet_p36       /home/ec2-user/anaconda3/envs/amazonei_mxnet_p36
amazonei_tensorflow_p27     /home/ec2-user/anaconda3/envs/amazonei_tensorflow_p27
amazonei_tensorflow_p36     /home/ec2-user/anaconda3/envs/amazonei_tensorflow_p36
chainer_p27              /home/ec2-user/anaconda3/envs/chainer_p27
chainer_p36              /home/ec2-user/anaconda3/envs/chainer_p36
mxnet_p27                /home/ec2-user/anaconda3/envs/mxnet_p27
mxnet_p36                /home/ec2-user/anaconda3/envs/mxnet_p36
python2                  /home/ec2-user/anaconda3/envs/python2
python3                  /home/ec2-user/anaconda3/envs/python3
pytorch_p27              /home/ec2-user/anaconda3/envs/pytorch_p27
pytorch_p36              /home/ec2-user/anaconda3/envs/pytorch_p36
tensorflow_p27           /home/ec2-user/anaconda3/envs/tensorflow_p27
tensorflow_p36           /home/ec2-user/anaconda3/envs/tensorflow_p36

Verify installed kernels

To verify installed kernels on the notebook, run the following command:

sh-4.2$ ipython kernelspec list

Check the installed version

To check the version that's installed in the conda environment, run the following command in the notebook instance terminal:

(python3) sh-4.2$ pip freeze | grep pandas

Or, run the following command in a notebook cell:

import pandas as pdpd.__version__

Create a persistent conda environment

If you stop a notebook, then Amazon SageMaker terminates the notebook's Amazon Elastic Compute Cloud (Amazon EC2) instance. Packages that you install in the conda environment don't persist between sessions. The only path that persists between notebook instance sessions is the /home/ec2-user/SageMaker directory for the notebook's Amazon Elastic Block Store (Amazon EBS) volume.

To make your libraries persist between sessions, see How can I be sure that manually installed libraries persist in Amazon SageMaker if my lifecycle configuration times out when I try to install the libraries?

Related information

Install external libraries and kernels

Customize a SageMaker notebook instance using a LLC script