- Newest
- Most votes
- Most comments
Hello,
As the partition columns are also written in the schema of the Parquet files, because of this when we read the data using DynamicFrame and perform some Spark action to the created DynamicFrame it fails with below error:
AnalysisException: Found duplicate column(s) in the data schema and the partition schema:
In order to fix this error, the ideal way is to fix the underlying Parquet files by re-writing them after dropping the partition columns.
However, as a workaround in this scenario, you can consider to update the Partition Column names manually in the Glue Data Catalog. Kindly refer [1] for more details. Then, you would be able to perform the action. By doing this, the partition column name read by the DynamicFrame will be referred from the Glue Data Catalog (as you are creating DynamicFrame using 'from_catalog()' method).
Thank you!
Reference:
[1] https://docs.aws.amazon.com/glue/latest/dg/console-tables.html#console-tables-details
Relevant content
- asked 2 years ago
- asked a year ago
- asked a year ago
- AWS OFFICIALUpdated 4 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 3 years ago