How to change data type in glue dataset catalogue

0

I managed to use glue crawler to crawled data (parquet file) from s3, however the column with type "boolean" is recognised as "string" when checking the data schema. Although i can edit the schema on the metadata catalogue, this is breaking the query on Athena as the data type is not recognised as boolean. Is there anyway to configure the serde parameter for glue crawler to automatically recognise this?

Additional information:

  • I have created the table using python pandas. Had workaround converting boolean to string given "True/False" is not recognised by glue/Athena.
  • Tried create table using Athena to ingest from s3. However, no data was shown when querying,
  • Tried visual ETL. Can't transform string to boolean.

Any suggestion would be appreciated.

Andy
preguntada hace un mes529 visualizaciones
1 Respuesta
1

Alternatively, you can try using the XMLClassifier provided by Glue. This Classifier is often better at inferring data types than the default BuiltInClassifiers. To use the XMLClassifier, select it in the Classifiers section when creating or updating your Glue Crawler.

Sources:

profile picture
EXPERTO
respondido hace un mes

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas