Sagemaker Lifecycle configuration

0

Trying to create a lifecycle config for notebook instance. error message Also there is nothing in cloudwatch logs corresponds to this failure.

asked 2 years ago952 views
2 Answers
0

From the screenshot, it looks like your LCC is taking longer than 5 minutes. SageMaker notebooks don't support scripts that run longer than 5 minutes (https://docs.aws.amazon.com/sagemaker/latest/dg/notebook-lifecycle-config.html). You can use the nohup command to run scripts longer than 5 minutes (see an example here - https://aws.amazon.com/premiumsupport/knowledge-center/sagemaker-lifecycle-script-timeout/#:~:text=If%20a%20lifecycle%20configuration%20script,ignores%20the%20hang%20up%20signal.)

Ideally, you should see a CW log stream in log group /aws/sagemaker/NotebookInstances in the format [notebook-instance-name]/[LifecycleConfigHook]. If you don't see it, make sure your notebook has the necessary permissions to write logs to CloudWatch, and has internet access/VPC endpoint to reach CloudWatch.

AWS
Durga_S
answered 2 years ago
  • I tried nohup command also in script but still getting same error.

  • Are you able to view CW logs? Can you share the script (if it's from our samples)? Another recommendation is to run the LCC shell script from the terminal, and see if it succeeds and how long it takes. That will give us an idea of whether it's an issue with the script or with the execution time.

0

Based on the name of the lifecycle in the error I'm assuming it's related to https://github.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/tree/master/scripts/auto-stop-idle , is that correct? I've been recently found that the .sh file used has some issues when running on aws2 notebooks. If you look in the logs you may see that it doesn't identify the python version being run then it exits the lifecycle. https://github.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/issues/109

I still haven't found any documented differences in how python is used/loaded, but I've tried setting up the idle function a few different ways. Running though the scrips manually after the notebook is up will work. If all else fails I'm looking at just loading the needed files in the lifecycle then setting a scheduled task to finish the setup. I know that's not as nice but if I can't find anything else that's where I'm heading.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions