Multidata architecture help

Question

How can I setup the datapipeline to help us setup a way to query 30-60 like databases.

Currently we are using a connector to replicate data from all our locations to RDS MySQL db.
We are wanting to use Quicksights to query Sets of data from all the RDS Sources brought into the  RDS MySQL db.

Would love some guidance on how we can accomplish this. I appreciate all of you brilliant people out there

Answer

Currently there may not be a way to accomplish this with DataPipeline. Instead, I would recommend using Glue Crawlers to crawl all the tables in RDS database - after which the table definitions will be stored in Glue Data Catalog. Then, you can access these tables from Athena, where you can perform queries and also implement some transformations using Glue DataBrew and then visualize them/create BI reports in QuickSight. 
See documentation references - 
1. https://docs.aws.amazon.com/glue/latest/dg/populate-data-catalog.html
2. https://docs.aws.amazon.com/athena/latest/ug/glue-athena.html
3. https://aws.amazon.com/blogs/big-data/data-preparation-using-an-amazon-rds-for-mysql-database-with-aws-glue-databrew/

Multidata architecture help

Contenuto pertinente