Skip to content

JSON timestamp.formats'='yyyy-MM-dd'T'HH:mm:ss.SSS'Z' returns HIVE_BAD_DATA: Error Parsing a column in the table: Cannot create timestamp, parsing error

0

Trying to have a Glue Crawler/Athena automatically pick up a timestamp format of yyyy-MM-dd'T'HH:mm:ss.SSS'Z'.

I have confirmed all the values in my dataset are formatted as this format, example: 2024-08-23T03:01:49.186Z. No values are null, every value in the data set has this format.

I have attempted to set the timestamp.formats property as both a SerDe and a table property, still receive the error.

Most of the posts I find related to HIVE_BAD_DATA seem to be about parsing specific values, but I receive an error for parsing the column and creating the timestamp. Is this an issue with my JSON formatting?

asked a year ago181 views
1 Answer
0

When creating your table in Athena, you should define the column as a STRING type initially. This is because the format you're using includes the 'T' and 'Z' characters, which are not part of the standard timestamp format that Athena recognizes automatically. (Refer - https://docs.aws.amazon.com/athena/latest/ug/data-types-timestamps.html)

Once the time column shows up as STRING, use can use date_parse() function to read it as a timestamp.

Below is an example of Athena SQL and it's output using the typeof() function

SELECT typeof('2024-08-23T03:01:49.186Z');
OUTPUT: varchar(24)

SELECT typeof(date_parse('2024-08-23T03:01:49.186Z', '%Y-%m-%dT%H:%i:%s.%fZ')) 
OUTPUT: timestamp(3)

SELECT date_parse('2024-08-23T03:01:49.186Z', '%Y-%m-%dT%H:%i:%s.%fZ') AS parsed_timestamp
OUTPUT: 	
2024-08-23 03:01:49.186

AWS
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.