How could we have Glue to get data from csv as String?

0

I have csv data uploaded to S3 bucket, letting Glue to set them up as tables for later use. I want all the columns loaded as string without designating each column name by name. How could we configure Glue to load all columns as string, not transforming into bigint or so?

demandé il y a 2 ans2003 vues
1 réponse
0
Réponse acceptée

Hello,

In Glue we use crawlers to automatically detect the schema from file and create a table in Glue catalog. For CSV files, the crawler reads either the first 100 records or the first 1 MB of data, whatever comes first to detect the schema. [1]

Having said that with this approach it is not possible to load all csv columns as string in Glue catalog directly. You can consider two approach for your use case:

  1. Create a crawler and run on csv data. Once it create the table in Glue catalog with correct datatype , you can modify the table schema to string for all columns.

  2. Directly read the data from csv files using Glue ETL job and in applymapping change schema to string and write the table into catalog with enableUpdateCatalog option. [2]

--Reference:

[1] https://aws.amazon.com/premiumsupport/knowledge-center/glue-crawler-detect-schema/ [2] https://docs.aws.amazon.com/glue/latest/dg/update-from-job.html

AWS
répondu il y a 2 ans

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions