Importing AWS Glue Data Catalog Database to SageMaker Studio Unified Catalog
I have an existing AWS Glue Data Catalog database that contains metadata for various datasets used in my data analytics and machine learning workflows. I want to import this database into the Amazon SageMaker Studio Unified Catalog and make it available within my SageMaker project. Please provide step-by-step instructions on how to accomplish this task, covering the following points:
Prerequisites:
- What permissions or IAM roles are required to perform this import?
- Are there any specific settings needed in the Glue Data Catalog or SageMaker Studio?
Validation and Troubleshooting:
- How can I verify that the import was successful?
- What should I do if I encounter any errors during the import process?
Adding to SageMaker Project:
- Once imported, how do I add the Glue database to my specific SageMaker project?
- Are there any additional steps required to make the data accessible within my project notebooks or pipelines?
Best Practices:
- Are there any best practices or considerations I should keep in mind when working with imported Glue databases in SageMaker Studio?
- How can I ensure that any updates to the original Glue Data Catalog are reflected in the SageMaker Studio Unified Catalog?
Limitations:
- Are there any limitations or restrictions when working with imported Glue databases in SageMaker Studio compared to the native Glue environment?
Please provide detailed answers and, if possible, include any relevant AWS CLI commands or Python code snippets that might be helpful in automating this process.
- Language
- English
asked 2 years ago1.6K views
- Newest
- Most votes
- Most comments
To import an AWS Glue Data Catalog database into the Amazon SageMaker Studio Unified Catalog and make it available within your SageMaker project, follow these steps:
Prerequisites:
- Ensure you have the necessary permissions to access both AWS Glue and Amazon SageMaker Studio. The exact IAM role requirements are not specified in the provided information, but you'll likely need read access to the AWS Glue Data Catalog and write access to SageMaker Studio.
- The source database must already exist in AWS Glue before you can import it.
Import Process:
- Navigate to Amazon SageMaker Unified Studio.
- Select your project.
- Go to "Data sources" under the Project catalog section.
- Create a new data source.
- Configure the data source:
- Provide a name and description
- Set the data source type as AWS Glue
- Configure optional connection settings
- Specify the AWS Glue database and table selection criteria
- Choose publishing settings
- Select metadata generation methods
- Configure data quality options
- Add metadata forms if desired
- Set run preferences (on-demand or scheduled)
- Review and create the data source
Validation and Troubleshooting:
- To verify the import was successful, check the project inventory in SageMaker Studio. The assets from the source database should be added there.
- If you encounter errors, review the configuration settings and ensure you have the necessary permissions. The provided information doesn't offer specific troubleshooting steps.
Adding to SageMaker Project:
- Once imported, the Glue database assets are automatically added to your project's inventory.
- To make the data discoverable by all domain users, you need to explicitly publish the inventory assets to the Amazon SageMaker Unified Studio catalog.
Best Practices:
- Curate your inventory assets with business metadata such as business names, descriptions, readme files, glossary terms, and metadata forms.
- To ensure updates to the original Glue Data Catalog are reflected in SageMaker Studio, you may need to run the data source again or set up a schedule for automatic updates.
- Only the latest published version of an asset is active in the discovery catalog. If you update an inventory asset after publishing, you must publish it again for the changes to appear.
Limitations:
- The provided information doesn't specify any particular limitations when working with imported Glue databases in SageMaker Studio compared to the native Glue environment.
Remember that Amazon SageMaker Unified Studio is a preview release service, so features and processes may change. Always refer to the most up-to-date AWS documentation for the latest information and best practices.
Sources
Create an Amazon SageMaker Unified Studio data source for AWS Glue in the project catalog - Amazon SageMaker Unified Studio
Amazon SageMaker Unified Studio data catalog - Amazon SageMaker Unified Studio
Amazon SageMaker Unified Studio terminology and concepts - Amazon SageMaker Unified Studio
This is so extremely generic as to be useless. I have followed these generic instructions and I still cannot see the database or any of the tables in sagemaker unified studio.