By using AWS re:Post, you agree to the Terms of Use

HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://bucket_1/parquet_sample/data_10720000_1 (offset=0, length=16981465): org.apache.parquet.io.GroupColumnIO cannot be cast

0

I converted a couple of compressed json files into parquet files with default snappy compression. The resulting files total less than 50MB. I am not sure why Athena throws this error. I read the background behind the error but I don't understand how Athena is querying several thousands of files which result in this error. I can run a simple select count(*) from table but a select * from table query fails

The json file had to be read line by line due to trailing error I encountered in Python.

Any idea why this occurs?

asked 22 days ago28 views
2 Answers
1
Accepted Answer

Hi There

I don't think this is related to the S3 rate-limiting topic that comes up when you search this error. You would see something like "Slow Down" if that was the case. The key is in the last part of the error

org.apache.parquet.io.GroupColumnIO cannot be cast

This error is probably caused by a parquet schema mismatch. Check your table creation query and test with a smaller subset of the data. There may be some issues with the data format or the table config. see https://docs.aws.amazon.com/athena/latest/ug/troubleshooting-athena.html

profile picture
answered 18 days ago
0

Thanks for the response!

answered 2 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions