How to customize crawlers to record Glue Job Target metadata

Question

I am looking to have metadata produced by crawlers during Glue Jobs record both source AND target information on the same data catalog table. From the research I have done, recording source metadata is native functionality, while configuring the crawler to record target information requires customization. What are the recommended steps to complete this customization?

Answer

Hiya, i think these recommendations can help you, Create a new crawler in the AWS Glue console.

In the "Crawler properties" section, select the "Target" tab.
In the "Target table" field, enter the name of the data catalogue table where you want to store the target metadata.
In the "Target database" field, enter the name of the database where the data catalogue table resides.
In the "Target table properties" section, select the "Include target information" checkbox.
In the "Target schema" section, select the schema that you want to use for the target metadata.
Click the "Save" button.

this should work

How to customize crawlers to record Glue Job Target metadata

Relevanter Inhalt