Cross-Account Deployment for SageMaker Unified Studio ETL Jobs

To deploy ETL jobs and orchestrators from one AWS account to another using Amazon SageMaker Unified Studio, you can implement a cross-account CI/CD pipeline approach. Here are the recommended practices:

Account Association Setup

1. From your development account (Account A), associate your SageMaker Unified Studio domain with your production account (Account B).
2. In Account B, accept the association request from Account A and enable the appropriate environment blueprints.
3. In Account A, create a custom project profile that specifies Account B's account ID and region under the Default tooling blueprint deployment settings.

Cross-Account Access Configuration

For your CI/CD pipeline to work across accounts, you'll need to configure proper IAM roles:

1. Create a pipeline role in Account A that can assume cross-account roles.
2. Create a cross-account role in Account B that can be assumed by Account A's pipeline role.
3. Configure appropriate permissions for these roles to access necessary resources like S3 buckets, Glue databases, and SageMaker resources.

Pipeline Implementation

You can use AWS CodePipeline to orchestrate the deployment process:

1. Set up a pipeline in Account A that's triggered when changes are committed to your ETL job repository.
2. Include stages for:

• Building any required containers and pushing them to ECR
• Running CloudFormation scripts to set up resources in Account B
• Deploying the ETL jobs to Account B

Resource Management

For ETL jobs that need to access data catalogs across accounts:

1. When creating projects in Account A, use the project profile that references Account B.
2. This allows your ETL jobs to read databases and tables from Account B as if they were in the same account.

Automation Components

To automate the deployment process:

1. Use AWS CDK or CloudFormation templates to define your infrastructure.
2. Set environment variables for account IDs, regions, and other configuration parameters.
3. Deploy stacks that create the necessary roles, S3 buckets, and other resources required for cross-account operations.

By following these practices, you can establish a repeatable, automated process for deploying ETL jobs from your development environment to production while maintaining proper separation between accounts.
Sources
Automation pipelines - Build a Secure Enterprise Machine Learning Platform on AWS
Cross account Glue Catalog in Amazon SageMaker Unified Studio | AWS re:Post
Build unified pipelines spanning multiple AWS accounts and Regions with Amazon MWAA | AWS Big Data Blog
Cross account Glue Catalog in Amazon SageMaker Unified Studio Domain | AWS re:Post
Streamline custom environment provisioning for Amazon SageMaker Studio: An automated CI/CD pipeline approach | AWS Machine Learning Blog