- Newest
- Most votes
- Most comments
This error typically occurs when Athena encounters issues reading Parquet files. Based on your description of the problem suddenly appearing with files that were previously working, there are a few potential causes:
-
In Athena engine version 3, there are stricter type validations that check compatibility before reading values. If you're using columns with mismatched data types (for example, a column defined as varchar in your table schema but stored as a different type in the Parquet file), this could cause failures. This is particularly relevant if you recently upgraded your Athena engine version.
-
When all values in a column are NULL, there might be an issue with how the Parquet statistics are being interpreted. This can sometimes cause Athena to fail when reading the file.
-
The Parquet file statistics might be causing issues. A common workaround is to set the Parquet SerDe property 'parquet.ignore.statistics' to 'true' using either:
- ALTER TABLE statement:
ALTER TABLE your_table_name SET TBLPROPERTIES ('parquet.ignore.statistics'='true') - Or when creating a new table with the SerDe property explicitly set
- There could be a data type mismatch between what's in your Parquet files and what's defined in your table schema. For example, if a column contains NULL values but the data type in the schema doesn't match the expected type in the Parquet file.
If the issue started suddenly without changes to your data or schema, it's possible that an Athena service update or configuration change is now enforcing stricter validation on your existing files.
Sources
Athena engine version 3 - Amazon Athena
Parquet SerDe - Amazon Athena
field data type INT32 is incompatible with type varchar defined in table schema, when selecting random samples | AWS re:Post
AWS Athena - HIVE_CURSOR_ERROR: incorrect data check | AWS re:Post
answered a year ago
Relevant content
- AWS OFFICIALUpdated 4 years ago
