1回答
- 新しい順
- 投票が多い順
- コメントが多い順
1
Hi,
AWS Glue Crawlers are used to automatically discover the schema of the data in Amazon S3 or other data sources. They also help in capturing schema evolution.
If your schema is fixed (do not change often), already known and you do not have issues creating your tables manually via the console or your code using the APIs, then you do not need to use them.
Consider also that the Crawler do have a cost, so cost optimization might be another reason to not use them if you are fine with self managing the schemas of your datasets.
for additional information on Crawlers, you can refer to this section of the AWS Glue Documentation.
hope this helps
関連するコンテンツ
AWS公式更新しました 3年前
As Fabrizio said correctly, You only need to run the AWS Glue Crawler again if your schema changes. Also, ETL job support bookmarking and its recommended to use when your data grows day by day and when your job runs, you don't want to perform ETL job operations all over your data again, with bookmarking it will start processing operation on new data (wont perform operation on a processed data).
Read more at: https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html