By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Help creating Source to Target Mapping Report using Glue that describes the mapping and behavior of existing Glue Jobs.

0

I am using glue to move data from s3 and redshift into a data lake. I would like to use a combination of AWS glue and Athena to create a source to target mapping report. The goal is to make the report dynamic, so that if an element in the ETL process is changed (i.e. a table name is altered) the report will adjust accordingly to the respective changes that have been made. What is the best practice that has been used to complete this task? Or what solutions exist that help solve this problem.

Topics
AnalyticsDatabase
Tags
Amazon AthenaAnalyticsAWS GlueExtract Transform & Load DataAmazon Redshift
Language
English
Andrew
asked 2 years ago403 views
1 Answer
0

If it's just the table name, just a job argument and then use the argument in the code.
For something more sophisticated you probably need some parameters and code logic, or just build a SparkSQL query dynamically.

It's not common practice to combine Glue and Athena for the same task (e.g. mapping and saving the data), in this case it sounds like Glue is more suited for that ETL.

profile pictureAWS
EXPERT
Gonzalo Herreros
answered 2 years ago
  • Andrew
    2 years ago

    Glue natively stores the source meta data in glue catalog. Is there a way to customize the crawlers to record target meta data in the same table? If so, what are the steps to do so?

  • Gonzalo Herreros EXPERT
    2 years ago

    You could use a crawler and then add some of your own metadata, but if you run the table again it can be overriden

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content