By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

How to change data type in glue dataset catalogue

0

I managed to use glue crawler to crawled data (parquet file) from s3, however the column with type "boolean" is recognised as "string" when checking the data schema. Although i can edit the schema on the metadata catalogue, this is breaking the query on Athena as the data type is not recognised as boolean. Is there anyway to configure the serde parameter for glue crawler to automatically recognise this?

Additional information:

  • I have created the table using python pandas. Had workaround converting boolean to string given "True/False" is not recognised by glue/Athena.
  • Tried create table using Athena to ingest from s3. However, no data was shown when querying,
  • Tried visual ETL. Can't transform string to boolean.

Any suggestion would be appreciated.

Topics
Analytics
Tags
Amazon AthenaAWS Glue
Language
English
Andy
asked a month ago512 views
1 Answer
1

Alternatively, you can try using the XMLClassifier provided by Glue. This Classifier is often better at inferring data types than the default BuiltInClassifiers. To use the XMLClassifier, select it in the Classifiers section when creating or updating your Glue Crawler.

Sources:

profile picture
EXPERT
Osvaldo Marte
answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content