glue job - Issues when reading from the glue catalog table using dynamic frame

0

Hi All, I have some issues when running my glue job, I landed my pipe delimited csv file in a s3 bucket and after running the crawler pointing to the folder where the file is placed, a glue catalog table is created.

However when I tried to read the data(code below) from the catalog table in a glue job for additional processing and converting to parquet, its not picking all the records.

dyf = glueContext.create_dynamic_frame.from_catalog(
            database=DATABASE,
            table_name=table_name,
            transformation_ctx="dyf-" + table_name,
    )
    rows = dyf.count()
    print(f"DataFrame records count : {rows}")

Please can someone suggest what could be the reason for the missing records? I see that there are three columns in the catalog table with incorrect data type( bigint in place of string). I went and manually corrected the data type and set the infer_schema = True in the above code. job is still not picking up the correct number of records.

  • How much is the difference? Is possible you have bookmarks enabled? How many do you get if you do spark.table(f"{DATABASE}.{table_name}").count()

  • Hi,

    Th Glue data Catalog is not a constraint, it is for information. You can force the schema using the glue resolve function. you have to manually force the schema

Pradeep
asked a year ago1800 views
1 Answer
0

Sounds your CSV has formatting issues, I would try to parse/validate it some other way to identify what's wrong, sometimes invisible characters can break the parsing but if it's missing records my guess is that some quote is not properly closed/escaped.

profile pictureAWS
EXPERT
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions