Glue Crawler - skip adding partitions
Partitioned data in S3 as below -
some-dataset/2019-12-28/
some-dataset/2019-12-29/
some-dataset/2019-12-30/
some-dataset/2019-12-31/
some-dataset/2020-01-01/
some-dataset/2020-01-02/
some-dataset/2020-01-03/
When running a Glue crawler against the above dataset to create the schema in the catalog, how can we configure the crawler to avoid adding partitions for year 2019 into the table?
If that is not possible, is there a way to totally avoid registering ANY partitions and only create the table schema?
- 최신
- 최다 투표
- 가장 많은 댓글
You can add excludes to the crawler, e.g. for anything 2019 "**/2019-**"
The second question doesn't make any sense to me, if you have a partitioned table you need at least one partition. You can disable the crawler once the table is created with at least one.
관련 콘텐츠
- 질문됨 한 달 전
AWS 공식업데이트됨 일 년 전- AWS 공식업데이트됨 일 년 전
- AWS 공식업데이트됨 일 년 전
