Glue Crawler error: Folder partition keys do not match table partition keys

Question

I have a glue crawler that continuously logs the following error:
"Folder partition keys do not match table partition keys, skipped folder: s3:////**year=2021/month=10**/"

It's the only path that doesn't get crawled amongst a dozen or so crawlers.

The structure for all files is like this:
"s3:////year=2021/month=10/day=01/file-abc.json.gz"

--but it seems the crawler stops at partition "month" and doesn't even go down to the "day" partition.

What I have checked:
- There are no files saved directly in the "month" partition.
- The structure is correct (month=10/day=01/file-xyz.json.gz etc)
- All files have correct content-type and content-encoding
- Deleting and re-uploading all files doesn't work
- Moving files to a different path (month=oct) doesn't work
- Deleting and creating a new crawler and classifier doesn't work
- Re-running countless times, same result
- Its not related to permissions (checked IAM, Lake Formation, SCP)

Configuration is as follows:
`
{
    "Version": 1,
    "CrawlerOutput": {
        "Partitions": {
            "AddOrUpdateBehavior": "InheritFromTable"
        },
        "Tables": {
            "AddOrUpdateBehavior": "MergeNewColumns"
        }
    },
    "Grouping": {
        "TableGroupingPolicy": "CombineCompatibleSchemas"
    }
}
 and SchemaChangePolicy 
{
    "UpdateBehavior": "UPDATE_IN_DATABASE",
    "DeleteBehavior": "DEPRECATE_IN_DATABASE"
}`

Any ideas to solve this are welcome!

Thanks

Answer

HI, please check the following:

* If the issue is caused by inconsistent partition structure, then make the structure consistent by renaming the S3 path manually or programmatically.

* If the partition is skipped due to mismatch in file format, compression format, or schema, and the data isn't required to be included in the intended table, then consider the following:
* Use an exclude pattern to skip any unwanted files.
* Move the unwanted file to a different location.

* If your data has different schemas in some input files and similar schemas in other input files, then combine compatible schemas when you create the crawler. On the Configure the crawler's output page, under Grouping behavior for S3 data (optional), select Create a single schema for each S3 path. When this setting is turned on and the data is compatible, then the crawler ignores the similarity of specific schemas when evaluating S3 objects in the specified include path. For more information, see How to create a single schema for each Amazon S3 include path.

Glue Crawler error: Folder partition keys do not match table partition keys

Relevant content