How do I troubleshoot issues with lifecycle configuration scripts in Amazon SageMaker AI?

Resolution

The LCC script times out

When your LCC script takes longer to run than the 5-minute quota, the script times out. 

To decrease the script runtime, take the following actions:

• Reduce the number of necessary steps. For example, limit the conda environments that you install large packages in.
• Run tasks in parallel processes.
• Run the nohup command in your script to ignore hangup signals so that the script can complete within 5 minutes.

To run the nohup command, initialize the command on the specific line of code, and then end the line of code with an ampersand.

Example script:

===
#!/bin/bash

set -e

# OVERVIEW
# This script executes an existing Notebook file on the instance during start using nbconvert(https://github.com/jupyter/nbconvert)

# PARAMETERS

ENVIRONMENT=python3
NOTEBOOK_FILE=/home/ec2-user/SageMaker/test.ipynb

source /home/ec2-user/anaconda3/bin/activate "$ENVIRONMENT"
nohup jupyter nbconvert --to notebook --ExecutePreprocessor.timeout=-1 --ExecutePreprocessor.kernel_name=python3 --execute "$NOTEBOOK_FILE" &

source /home/ec2-user/anaconda3/bin/deactivate

====

You can't delete existing LLC

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

When you try to delete an LCC script that's already attached to a SageMaker Studio domain, you might receive the following error message:

"Loading of domain lifecycle failed. [400] ResourceNotFound: LifecycleConfig does not exist."

To resolve this issue, you must first detach the LCC script, and then delete it.

Or, you can set the LifecycleConfigArns parameter in your domain configuration to an empty list.

To reconfigure your domain, complete the following steps:

1. Run the following describe-domain command to view the list of associated LCC scripts in the LifecycleConfigArns parameter:

   aws sagemaker describe-domain —domain-id d-xxxxxxxxx

2. Run the following update-domain command to set the parameter to an empty list:

   aws sagemaker update-domain --domain-id d-xxxxxxxxx \
   --default-user-settings '{
     "JupyterServerAppSettings": {
       "DefaultResourceSpec": {
         "InstanceType": "system"
       },
       "LifecycleConfigArns": []
     }
   }'

When you run the update-domain command again, the domain no longer references the LCC script. You can then attach the new LCC script to the domain.

The LCC script can't install a package

If the LCC can't install packages in a SageMaker Studio domain or notebook instance, then the LCC script might not have access to the domain or instance. To resolve this issue, you must connect the associated SageMaker Studio domain or notebook instance to Amazon Virtual Private Cloud (VPC). The VPC must also allow internet access.

You can either associate a VPC with default communication over the internet in PublicInternetOnly mode. Or, you can set up communication with the internet in VPC only mode.

If you use VPC only mode, then set up security groups with inbound and outbound rules that allow the following traffic:

• NFS traffic over TCP on port 2049 between the domain and the Amazon Elastic File System (Amazon EFS) volume.
• TCP traffic within the security group.
  Note: TCP traffic in the security group is required to connect the Jupyter Server application and the Kernel Gateway applications. You must allow access to at least ports in the 8192–65535 range.

To allow internet access, you must use a NAT gateway with access to the internet.

The instance can't find the LCC script content

When the application instance can't find the content that's in the LCC script, you might receive the following error message:

"stdbuf: failed to run command "/opt/ml/lifecycleconfig/lifecycle_script.sh": No such file or directory"

This error can occur because of the incompatibility between Windows formatted symbols and Unix formatted symbols. Window characters use CRLF line endings (\r\n), and Unix uses LF line endings (\n).

Amazon SageMaker AI notebook instances run on the Amazon Linux 2 (AL2) operating system (OS). An LCC bash script that you download on a Windows OS and attach directly to the LLC isn't valid for the Linux OS.

To resolve this issue, complete the following steps:

1. Open the bash script in Notepad++.
2. Choose View, and then choose Show symbol.
3. Choose Show all symbols to view the Windows (\r\n) line endings. 
4. Change CRLF to LF (\n) to make the script valid in Unix.
5. Attach the script to the LCC.

Related information

Debug lifecycle configurations

Connect Studio notebooks in a VPC to external resources