How do I troubleshoot Amazon SageMaker AI notebook instance timeout issues when I install additional libraries?

Resolution

When a lifecycle configuration script runs for longer than 5 minutes, the script fails, and SageMaker AI doesn't create or start the notebook instance.

Use one of the following methods to resolve the issue.

Run the nohup command

For a temporary solution, run the nohup command to force the lifecycle configuration script to continue to run in the background until you install the packages.

Run the following command with nohup at the beginning and an ampersand (&) at the end:

#!/bin/bash
set -e
nohup pip install xgboost &

After you install the libraries, the script stops running. SageMaker AI doesn't notify you when this happens, but you can run the ps command to check the status of the script.

Note: You can also run the nohup command for other script timeout scenarios, such as when you download large Amazon Simple Storage Service (Amazon S3) objects.

Create a custom persistent Conda installation on the notebook instance's volume

1. In the terminal of an existing notebook instance, run the following command in your preferred editor to create a .sh file:

   vim custom-script.sh

2. Add the contents of the on-create script to the .sh file. To copy the script, see amazon-sagemaker-notebook-instance-lifecycle-config-samples on the GitHub website. The script creates a new Conda environment in a custom Conda installation. The script also installs NumPy and Boto3 in the new Conda environment.
   Note: The notebook instance must have internet connectivity to download the Miniconda installer and ipykernel.

3. Run the following command to mark the script as executable and run the script:

   chmod +x custom-script.sh
   ./custom-script.sh

4. After the installation completes, stop the notebook instance.

5. Add the on-start script to the .sh file. To copy the script, see amazon-sagemaker-notebook-instance-lifecycle-config-samples on the GitHub website.

6. On the stopped notebook instance, add the on-start script as a lifecycle configuration. Each time you start the notebook instance, the script makes the custom environment available as a kernel in Jupyter.

7. Start the notebook instance, and then install your custom libraries in the custom environment.
   For example, to install pyarrow, run the following command:

   import sys
   !conda install --yes --prefix {sys.prefix} -c conda-forge pyarrow

If you stop and then start your notebook instance, then your custom Conda environment and libraries are still available. You don't need to install them again.

If you receive an error message that says that you must update Conda, then run the following commands:

!conda install -p "/home/ec2-user/anaconda3" "conda>=4.8" --yes
!conda install -p "/home/ec2-user/SageMaker/custom-miniconda/miniconda" "conda>=4.8" --yes

After the Conda installation completes, install the libraries.

Create a Neptune notebook and configure its policies and connections

When you create a notebook in Amazon Neptune, Neptune uses a default life cycle configuration that points to an S3 object, such as aws s3 cp s3://aws-neptune-notebook/graph_notebook.tar.gz /tmp/graph_notebook.tar.gz. The Neptune AWS Identity and Access Management (IAM) role must have a permissions policy that grants the s3:GetObject and s3:ListBucket actions for the arn:aws:s3:::aws-neptune-notebook and arn:aws:s3:::aws-neptune-notebook/ resources.

The Neptune IAM role must also have a trust policy that grants SageMaker AI the AssumeRole action. For example, you activated IAM database authentication on the DB cluster that you associated with the notebook. The IAM role must have a permissions policy that grants the neptune-db: action for the associated DB cluster.

If you use a virtual private cloud (VPC) to connect your Neptune notebook to the internet, then you must attach a NAT gateway to the VPC. Also, you must configure the security groups to allow internet access to Amazon S3 and the Python Package Index (PyPI). For more information about PyPI, see the Python Package Index website.

Note: To troubleshoot issues with lifecycle configuration scripts, use Amazon CloudWatch Logs. You can find the logs in the following locations:

• log group: /aws/sagemaker/NotebookInstances
• log stream: example_notebook_instance_name/LifecycleConfigOnStart

Related information

What is Amazon SageMaker AI?

Debug lifecycle configurations