By using AWS re:Post, you agree to the Terms of Use
/Run different notebooks present in same Sagemaker notebook instance using lifecycle configurations based on different lambda triggers/

Run different notebooks present in same Sagemaker notebook instance using lifecycle configurations based on different lambda triggers

0

I have a sagemaker notebook instance having two jupyter notebook ipynb files. When I had one jupyter notebook, I was able to run it automatically with one lambda function trigger and lifecycle configuration.

Now I have two jupyter notebooks and corresponding two lambda function triggers. How can I run them based on the trigger by changing the lifecycle configuration script.

The trigger is file uploading into S3. Based on what location the file is added, the corresponding jupyter notebook should run

1 Answers
1

For the use case of automatically running ML jobs with on-demand infrastructure, with the ability to accept input parameters, I'd recommend SageMaker Processing as a better fit than Notebook Instances + Lifecycle Configs.

With processing jobs:

  • You could still use notebook files if needed, using a tool like Papermill or a more basic pattern like just loading the file and running through the code cells. For example using a FrameworkProcessor, you should be able to upload a bundle of files to S3 (including your notebooks and a plain Python entrypoint to manage running them).
  • You could trigger processing jobs from events just like your current notebook start-up, but could provide many different parameters to control what gets executed.
  • The history of jobs and their parameters will be automatically tracked through the SageMaker Console - with logs and metrics also available for analysis.
  • You wouldn't be limited to the 5 minute time-out of a LCConfig script

If you really needed to stick with the notebook pattern though, modifying the LCConfig each time seems less than ideal... So maybe I'd suggest bringing in another external state you could use to manage some state: For example have your LCConfig script read a parameter from SSM or DynamoDB to tell it which notebook to execute on the current run?

EXPERT
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions