I'm trying to troubleshoot why the sagemaker staging endpoint is not using the latest model version. I did the following steps:
- I started a new Sagemaker project and used the build, train, deploy and monitor model template.
- The pipeline automatically executes upon creating the project and I approved that model. The endpoint was successfully created using this model version 1.
- I cloned the repo, made a inconsequential change (just changed some training hyper params) and pushed the changes. Pipeline executes again successfully creating model version 2.
- I approved the version 2 and deploy script also runs successfully it seems based on the deploy logs but the endpoint keeps using version 1 of the model. (Also for some reason version 1 of the model appears as "Model-xxxxxxx" in the Models list while version 2 appears as "Pipeline-xxxxxx-xxxxxx"
Would appreciate any help. Thank you!
Did you approve the transition to "production" in CodeBuild for the first model? In the deploy script, in the function that retrieves the latest approved model, can you print which model it finds and verify that it's finding version 2? Additionally, the deployment script prints out cloudformation templates, did you verify that it shows version 2? If you can paste the deploy logs here that would be helpful.