Sagemaker and Data on Databases
A customer has a question about data sources
“most of our data is stored in SQL databases, while the SageMaker docs say that I have to put it all in S3. It’s not obvious what the best way to do this is. I can think for example of splitting my analysis code in two; one pre-processing step to go from SQL queries to tabular data, and e.g. store that as Parquet files. For high-dimensional tensor data it’s even less obvious.”
Can someone comment on that?
- 최신
- 최다 투표
- 가장 많은 댓글
We have an example notebook for interacting from Redshift data from a SageMaker managed notebook, which I believe is suitable for an Exploratory Data Analysis (EDA) use-case: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/working_with_redshift_data/working_with_redshift_data.ipynb
For production purposes, the customer should consider separating the job of first extracting data from relational databases to S3 (to build out a data lake), and then using that for downstream processing/machine learning (including SageMaker, EMR, Athena, Spectrum, etc.). Customers can build extraction pipelines from popular relational databases using AWS Glue, EMR, or their preferred ETL engines like those on the AWS Marketplace.
I'd recommend using SageMaker Data Wrangler to connects the dots of different SageMaker services. https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-import.html
for custom models, you could use Import data into Canvas from database dircetly :Connect to data in Amazon S3, Amazon Athena, or Amazon RDS For Amazon RDS, if you have the AmazonSageMakerCanvasFullAccess policy attached to your user’s role, then you’ll be able to import data from your Amazon RDS databases into Canvas.
https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-importing-data.html
