I need to read S3 data, transform and put into Data Catalog. Should I be using a Crawler?

Question

Files are uploaded every hour to an S3 bucket.  I currently have a Glue ETL job reading the S3 bucket, transforming data and inserting into a Glue Data Catalog.  I have seen examples where people have a Glue Crawler which reads the S3, writes data to Data Catalog table, and then an ETL job reads from a table transforms and then writes back to another table (or wherever it needs to go).  Should I be using a Crawler? I don't see the need for it if I can just use the ETL job to go S3->Transform->Data Catalog.  It would seem the ETL job supports bookmarking (init/commit) just like Crawlers do.

Accepted Answer

Hi,

AWS Glue Crawlers are used to automatically discover the schema of the data in Amazon S3 or other data sources. They also help in capturing schema evolution.

If your schema is fixed (do not change often), already known and you do not have issues creating your tables manually via the console or your code using the APIs, then you do not need to use them.

Consider also that the Crawler do have a cost, so cost optimization might be another reason to not use them if you are fine with self managing the schemas of your datasets.

for additional information on Crawlers, you can refer to [this section of the AWS Glue Documentation](https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html).

hope this helps

I need to read S3 data, transform and put into Data Catalog. Should I be using a Crawler?

相关内容