Athena : serDe should not be accessed from a null StorageFormat
I'm stroing cloudtrail logs into the S3 and through glue crwaler trying to crawl the S3(cloudtrail logs) and creating a glue table. While I'm running query against that table in Athena , getting below error
HIVE_UNKNOWN_ERROR: serDe should not be accessed from a null StorageFormat
Can someone please help in fixing the issue or tell the workaround. I'm creating all the resources through terraform.
Check the table parameters for the SerdeInfo of the StorageDescriptor. The crawlers don't always detect and configure the InputFormat, OutputFormat and SerializationLibrary parameters, so they can be NULL. Ex:
$ aws glue get-table --region <region> --profile <profile> --database-name <db name> --name <table name> --query '*[].StorageDescriptor.[InputFormat,OutputFormat,SerdeInfo.[SerializationLibrary]]'
[ [ null, null, null ] ]
If you manually edit the table (and recreate any partitions), then you'll be cleared to query. Ex: (for a .csv file)
[ [ "org.apache.hadoop.mapred.TextInputFormat", "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat", "org.apache.hadoop.hive.serde2.OpenCSVSerde" ] ]
Relevant questions
How to escape a comma in a csv file in AWS Glue?
Accepted AnswerGlue to automatically create target schema
Accepted Answerasked 5 years agocrawled data can not be queried in athena
Accepted Answerasked 5 years agoHIVE_METASTORE_ERROR when running an Athena query to select the first 10 rows from a partitioned table created by a Glue Crawler.
asked a month agoGlue processing a csv
Accepted Answerasked 2 years agoGlue Jobs & Multiple tables
Accepted Answerasked 4 years agoGlue table not showing in console
asked a month agoHIVE_UNKNOWN_ERROR: serDe should not be accessed from a null StorageFormat
asked 2 years agoAthena : serDe should not be accessed from a null StorageFormat
asked 4 months agoRunning glue crawler on encrypted S3 objects present in different account
asked 4 months ago