By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

AWS Glue Loading into Redshift with empty values

0

I have a AWS Glue pyspark ETL job and I need to load a redshift table from the dataframe. Several date and string columns have NULL values in the DataFrame in a subset of rows. I am getting this error when trying to run the glueContext.write_dynamic_frame.from_catalog command in the glue pyspark script...

IllegalArgumentException: Don't know how to save NullType to REDSHIFT

I am not sure how to resolve as I cannot "fill" these missing values. They are simply optional dates and values that do not exist. I can find no documentation or information in AWS about how to resolve this issue.

Topics
AnalyticsDatabase
Tags
AWS GlueExtract Transform & Load DataAmazon Redshift
Language
English
johnclarity
asked 2 years ago1923 views
1 Answer
0

This error is to be expected when there are null values in the data since it seems like Redshift does not allow to write values with NullType. Some suggestions to avoid this error are -

  1. Remove the rows containing null values using Spark SQL by converting glue dynamic frame to spark dataframe and then deleting rows with null values.
  2. Use DropNullFields option in Glue - https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-transforms-DropNullFields.html
AWS
Meghana_S
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content