Skip to content

Incompatible Issue with timestamp

0

HIVE_BAD_DATA: The Data Field type BINARY in parquet file is incompatible with type timestamp(3) defined in table schema Can anyone help us to sort this issue out

asked 2 years ago499 views

1 Answer
1

This error indicates a mismatch between the data type in your Parquet file and the schema defined in your table. Specifically, the Parquet file contains a BINARY data type for a field that is defined as timestamp(3) in your table schema.

To resolve this issue, you have a few options:

  1. Update the table schema: If the data in the Parquet file is actually in a binary format, you may need to update your table schema to match. You could change the column type to BINARY or VARCHAR in your table definition, and then use appropriate casting or conversion functions when querying the data.

  2. Correct the data conversion: If the data should indeed be a timestamp, the issue may be in how the data is being converted to Parquet format. Ensure that your ETL process or data pipeline is correctly converting the timestamp data to the appropriate Parquet timestamp format.

  3. Use schema evolution: Parquet supports schema evolution, but you need to ensure that your conversion process handles this correctly. If you've changed the schema, make sure you're properly managing schema evolution in your data pipeline.

  4. Verify Parquet file integrity: Use tools like parquet-tools to inspect your Parquet files and verify that the schema and data types are as expected.

  5. Consider using a CTAS (Create Table As Select) query: This can help you create a new table with the correct schema while converting the data types as needed.

If these steps don't resolve the issue, you may need to investigate your data pipeline more closely to understand why the timestamp data is being stored as BINARY in the Parquet files. It's possible that there's an issue in how the data is being processed or converted before being written to Parquet format.
Sources
Hive Bad Data error | AWS re:Post
HIVE_BAD_DATA: Not valid Parquet file | AWS re:Post

answered 2 years ago

EXPERT

reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.