AWS Glue ETL Automation

Question

Our current setup involves AWS Glue in operation, where data is being extracted from one SQL Server and loaded into another SQL Server through use of AWS Glue Studio for selected tables.

Is there a way to configure AWS Glue to monitor changes in source table columns while maintaining the existing data transfer process, and additionally, update the table columns in the destination SQL Server before transferring the data? How would we automate this?

Answer

This is possible but you would have to use the script side of glue.

For example I’ve written a Python script to move data between servers mssql and Postgres. You would have to develop a script that compared the schemas between 2 database and update accordingly.

AWS Glue ETL Automation

Contenuto pertinente