Help creating Source to Target Mapping Report using Glue that describes the mapping and behavior of existing Glue Jobs.


I am using glue to move data from s3 and redshift into a data lake. I would like to use a combination of AWS glue and Athena to create a source to target mapping report. The goal is to make the report dynamic, so that if an element in the ETL process is changed (i.e. a table name is altered) the report will adjust accordingly to the respective changes that have been made. What is the best practice that has been used to complete this task? Or what solutions exist that help solve this problem.

1 Answer

If it's just the table name, just a job argument and then use the argument in the code.
For something more sophisticated you probably need some parameters and code logic, or just build a SparkSQL query dynamically.

It's not common practice to combine Glue and Athena for the same task (e.g. mapping and saving the data), in this case it sounds like Glue is more suited for that ETL.

profile pictureAWS
answered a year ago
  • Glue natively stores the source meta data in glue catalog. Is there a way to customize the crawlers to record target meta data in the same table? If so, what are the steps to do so?

  • You could use a crawler and then add some of your own metadata, but if you run the table again it can be overriden

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions