AWS GLue Spark job: Found duplicate column(s) in the data schema and the partition schema: `day`, `month`, `year`

Question

Glue spark job is failing with the error message: AnalysisException: Found duplicate column(s) in the data schema and the partition schema: day, month, year. And my actual parquet data file in S3 includes these partition columns as well.

Code snippet:
```
dyf= glueContext.create_dynamic_frame.from_catalog(database = db,table_name = tbl, additional_options={"catalogPartitionPredicate":"year>=2022 and month>=10 and day>=1"},transformation_ctx="dyf")
dyfdrop = dyf.drop_fields(paths=["year", "month", "day"])
dyfdrop.toDF().printSchema()
dyfdrop.toDF().show()
```

S3 Path: 
s3://bucket/f1/f2/tbl/year=2022/month=10/day=1/

Partition column names in glue catalog: "year", "month", "day"

I am reading from Glue catalog by filtering partition column using above code.

I have also tried to drop partition columns and still it didnt work.

Please help me to fix this in order to read these files.

Thanks

Answer

Hello,

As the partition columns are also written in the schema of the Parquet files, because of this when we read the data using DynamicFrame and perform some Spark action to the created DynamicFrame it fails with below error:

------------
	AnalysisException: Found duplicate column(s) in the data schema and the partition schema:
------------

In order to fix this error, the ideal way is to fix the underlying Parquet files by re-writing them after dropping the partition columns.

However, as a workaround in this scenario, you can consider to update the Partition Column names manually in the Glue Data Catalog. Kindly refer [1] for more details. Then, you would be able to perform the action. By doing this, the partition column name read by the DynamicFrame will be referred from the Glue Data Catalog (as you are creating DynamicFrame using 'from_catalog()' method).

Thank you!

Reference:

[1] https://docs.aws.amazon.com/glue/latest/dg/console-tables.html#console-tables-details

AWS GLue Spark job: Found duplicate column(s) in the data schema and the partition schema: `day`, `month`, `year`

관련 콘텐츠