My Amazon SageMaker notebook instance that's backed with an AWS Glue development endpoint times out with the following error:
"Notebook Instance Lifecycle Config 'arn:aws:sagemaker:us-xxxx-x:xxxxxx:notebook-instance-lifecycle-config/mylifecycleconfig' for Notebook Instance 'arn:aws:sagemaker:us-xxxx-x:xxxxxx:notebook-instance/mynotebook' took longer than 5 minutes. Please check your CloudWatch logs for more details if your Notebook Instance has Internet access."
This error might indicate that the Apache Livy service on the AWS Glue development endpoint can't communicate with the server where the notebook runs. The most common reason for this issue is that the lifecycle configuration attached to the notebook might reference a non-existing development endpoint. The SageMaker lifecycle script is used to configure the SparkMagic plugin to call the REST API against a Livy server running on a development endpoint. To learn how these components work together, see How AWS Glue development endpoints work with SageMaker notebooks.
To resolve this issue, try these troubleshooting options.
Create a new development endpoint
Create a new AWS Glue development endpoint, preferably with the same name and configuration as the endpoint that you were using. For more information, see Adding a development endpoint.
Update the lifecycle configuration of the notebook instance
You must update the lifecycle configuration of the notebook instance that failed if either of the following conditions are true:
- You created a development endpoint with a different name.
- You want to use a development endpoint that you've previously created.
To update the lifecycle configuration of the notebook, do the following:
1. Open the SageMaker console.
2. In the navigation pane, under Notebook, choose Lifecycle configurations.
3. Select the lifecycle configuration of the notebook instance that failed.
4. Choose Edit.
5. In the Scripts section, for both Start notebook and Create notebook scripts, replace the name of the development endpoint in the following command with that of the endpoint (example_new_dev_endpoint) that you created:
Be sure to update us-east-1 in the command with the Region of your AWS Glue development endpoint.
aws glue get-dev-endpoint --endpoint-name example_new_dev_endpoint --endpoint https://glue.us-east-1.amazonaws.com
Note: Be sure that the SageMaker notebook has the required permissions to access the development endpoint.
Recover the notebook data
You can consider recovering the notebook when you have access to SageMaker but can't create a new development endpoint.
To recover your notebook data, perform the following steps:
- Open the SageMaker console.
- In the navigation pane, under Notebook, choose Notebook instances.
- Select the notebook that you want to recover.
- If your notebook Status is not Stopped, choose Actions, and then choose Stop to stop the notebook instance.
You can see that the notebook instance status changes to Stopping. Wait until the status changes to Stopped to complete the rest of the steps.
- Choose Actions, and then choose Update settings.
- In the Edit notebook instance page, under Additional Configuration, for Lifecycle configuration, select No configuration.
- Choose Update Notebook Instance.
Your notebook instance status changes to Updating. When the notebook instance update is complete, the status changes to Stopped. Choose Start to start the instance. After the instance is started, you can access Jupyter and JupyterLab to download your scripts and data locally. However, you still can't connect to the AWS Glue development endpoint.
Important: You can link your scripts to a GitHub repository to avoid the loss of data in the future.
Note: If the lifecycle configuration times out when you try to install the libraries, then see How can I be sure that manually installed libraries persist in Amazon SageMaker if my lifecycle configuration times out when I try to install the libraries?
Tutorial: Use a SageMaker Notebook with your development endpoint
SageMaker notebooks on the AWS Glue console
Working with notebooks on the AWS Glue console
Create an IAM policy for SageMaker notebooks