- Newest
- Most votes
- Most comments
This error indicates a mismatch between the data type in your Parquet file and the schema defined in your table. Specifically, the Parquet file contains a BINARY data type for a field that is defined as timestamp(3) in your table schema.
To resolve this issue, you have a few options:
-
Update the table schema: If the data in the Parquet file is actually in a binary format, you may need to update your table schema to match. You could change the column type to BINARY or VARCHAR in your table definition, and then use appropriate casting or conversion functions when querying the data.
-
Correct the data conversion: If the data should indeed be a timestamp, the issue may be in how the data is being converted to Parquet format. Ensure that your ETL process or data pipeline is correctly converting the timestamp data to the appropriate Parquet timestamp format.
-
Use schema evolution: Parquet supports schema evolution, but you need to ensure that your conversion process handles this correctly. If you've changed the schema, make sure you're properly managing schema evolution in your data pipeline.
-
Verify Parquet file integrity: Use tools like parquet-tools to inspect your Parquet files and verify that the schema and data types are as expected.
-
Consider using a CTAS (Create Table As Select) query: This can help you create a new table with the correct schema while converting the data types as needed.
If these steps don't resolve the issue, you may need to investigate your data pipeline more closely to understand why the timestamp data is being stored as BINARY in the Parquet files. It's possible that there's an issue in how the data is being processed or converted before being written to Parquet format.
Sources
Hive Bad Data error | AWS re:Post
HIVE_BAD_DATA: Not valid Parquet file | AWS re:Post
Relevant content
asked a year ago
asked 10 years ago
asked 2 years ago
