Glue Crawler - skip adding partitions

Question

Partitioned data in S3 as below -

some-dataset/2019-12-28/

some-dataset/2019-12-29/

some-dataset/2019-12-30/

some-dataset/2019-12-31/

some-dataset/2020-01-01/

some-dataset/2020-01-02/

some-dataset/2020-01-03/

When running a **Glue crawler** against the above dataset to create the schema in the catalog, how can we configure the crawler to **avoid adding partitions** for year 2019 into the table?

If that is not possible, is there a way to totally avoid registering ANY partitions and only create the table schema?

Accepted Answer

You can add excludes to the crawler, e.g. for anything 2019 ```"**/2019-**" ```
The second question doesn't make any sense to me, if you have a partitioned table you need at least one partition. You can disable the crawler once the table is created with at least one.

Glue Crawler - skip adding partitions

Contenuto pertinente