AWS Athena generic internal error when querying a table created using a Glue Crawler which says 'columns has 110 elements while columns. Types has 106 elements!'

0

First of all, I have already asked about this problem here in re:Post and on Database Administrators Stack Exchange here on Monday, but I received no follow up when the first "Answer" here provided no solution for me in terms of what to do and my question on DASE has no answer attempts yet and the only comment has since been removed and I really need a solution ASAP. My masters degree depends on it to be candid!

This error happens to me anytime I try to query the 'airports' table in my database which was created by a glue crawler I created for this purpose along with 3 other datasets. They were all csv files beforehand, and I have performed each and every step of this process already in MySQL Workbench locally before trying it out on the cloud in AWS right now and I never got any errors like this when querying my airports dataset there. All I did was run this query:

SELECT "loc id" FROM "runway_db_athena"."airports" limit 10;

And this is the error message I got back:

GENERIC_INTERNAL_ERROR: SerDeException thrown initializing deserializer org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe. Cause: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe: columns has 110 elements while columns.types has 106 elements! This query ran against the "runway_db_athena" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: d9e95a16-9399-434b-8c19-88591927fc6b

What does this mean and how exactly do I fix it? I want step by step instructions please.

p.s. As a sanity check before re-posting here after the "Answer" on my previous question here indicated that this error typically means you have a csv with more fields than the ones identified by the crawler, so, I opened up the file, hit Ctrl+-> and saw that there are 106 columns, then I opened up MySQL Workbench, created a new database, and imported this csv file using its Data Table Import Wizard, then ran a DESC and a SELECT on it and both worked just fine. Plus, Workbench triple confirmed that the 110 was the off number:

3   21  04:08:29    desc `airportandrunwaydata`.airports    106 row(s) returned 0.000 sec / 0.000 sec

Enter image description here

For the record, the other 3 csv files which were gone over by the same Glue Crawler all had tables created for them which work completely without issue.

profile picture
asked a year ago116 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions