SageMaker Pipelines and CI/CD with GitLab Multiaccount

0

Hi,

I have a scenario where I am building different models for multiple different use cases. For each use case I have a SageMaker Pipeline for it ( with preprocessing, training ....deployment etc). Thus I have multiple independent pipelines.

Note : Using workflows and steps

How to do Sagemaker pipelines CI/CD across multiple account using GitLab?

Limitations/ Things which I cannot use/allow due to compliance/policy :

  • Cannot use MLOps template directly given by AWS : As sagemaker starts creating repo.
  • dev, test and prod all are in 3 different account
  • cannot use aws CodeCommit, CodePipeline ( can only use GitLab)

Note : Few use case requires continuous training and batch transforms.

Any suggestions would be appreciated.

Thanks in advance.

2개 답변
0

The main steps to deploy models in several accounts would be --> Create a model --> register model in model registry creating a model group then attach two policies, one for the S3 bucket containing the model.tar.gz and the second policy for the model group. In the target account you create a model pointing to the above model package group. And finally you simply deploy the model with the model you have just created in the target account. If you want to see the process in detail please refer to the blog below:

https://aws.amazon.com/blogs/machine-learning/multi-account-model-deployment-with-amazon-sagemaker-pipelines/

AWS
전문가
답변함 2년 전
0

I'm assuming you want to run all the SageMaker Pipelines in one account and deploy the models to a different account. In that case you need to do a few things:

  1. Allow the target accounts to read the models in the S3 bucket of the main account;
  2. Attach a policy to the ModelPackageGroup (SageMaker Pipelines) that allows the target accounts to read it;
  3. Create an EventBridge rule that detects events (Approval) in the ModelPackageGroup and invoke GItLab to start the CI/CD process that will:
  • Create a new Model in the SageMaker Models Catalog of the target accounts that points to the Version you just registered in the ModelPackageGroup of the main account
  • Finally deploy the model to the target account by using the Model you just created in each target account.

If you need to run each SageMaker pipeline in a different account, I would just use GitLab to, kick-off the pipelines using SageMaker Python Library or Boto3.

Regarding the template, you can also create a custom SageMaker Project template the way you need, instead of using the MLOps templates: https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-projects-templates-custom.html

In this blog post you have an example of how to create the policies for S3 and ModelGroup. You can this blog as an inspiration: https://aws.amazon.com/blogs/machine-learning/multi-account-model-deployment-with-amazon-sagemaker-pipelines/

AWS
답변함 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠