Run different notebooks present in same Sagemaker notebook instance using lifecycle configurations based on different lambda triggers

For the use case of automatically running ML jobs with on-demand infrastructure, with the ability to accept input parameters, I'd recommend SageMaker Processing as a better fit than Notebook Instances + Lifecycle Configs.

With processing jobs:

• You could still use notebook files if needed, using a tool like Papermill or a more basic pattern like just loading the file and running through the code cells. For example using a FrameworkProcessor, you should be able to upload a bundle of files to S3 (including your notebooks and a plain Python entrypoint to manage running them).
• You could trigger processing jobs from events just like your current notebook start-up, but could provide many different parameters to control what gets executed.
• The history of jobs and their parameters will be automatically tracked through the SageMaker Console - with logs and metrics also available for analysis.
• You wouldn't be limited to the 5 minute time-out of a LCConfig script

If you really needed to stick with the notebook pattern though, modifying the LCConfig each time seems less than ideal... So maybe I'd suggest bringing in another external state you could use to manage some state: For example have your LCConfig script read a parameter from SSM or DynamoDB to tell it which notebook to execute on the current run?

How about putting two notebooks in two different notebook instances?