Unsupported case of DataType

1

ERROR MESSAGE: An error occurred while calling o518.pyWriteDynamicFrame. Unsupported case of DataType: com.amazonaws.services.glue.schema.types.StringType@235d3a6f and DynamicNode: integernode.

I utilize glue catalog and use s3 as my DB. I'm uploading CSV files and processing them with glue jobs, however I continue to get the error message **"An error occurred while calling o518.pyWriteDynamicFrame. Unsupported case of DataType: com.amazonaws.services.glue.schema.types.StringType@235d3a6f and DynamicNode: integernode.Datatype". **

I've updated and recreated the glue tables and crawlers multiple times as well as the schema to match data types but I still get the same error message. The glue job only succeeds when I update the schema to have string as the only data type but the data type needs to be int for integers when querying in Athena so this will cause an issue down stream. Can anyone help?

  • I had the same problem. Not sure about the cause but I was able to circumvent the error by using

    format="parquet"

    instead of

    format="glueparquet"

    in my glueContext.write_dynamic_frame.from_options() call

    This is probably slower, but that's not a problem in my case.

1 Answer
0

To troubleshoot, Inspect your columns with the string data type and convert any string to integer. Refer to the considerations and limitations about using SQL queries in Athena. If the problem persists, please contact AWS Support. For more information on how to gather troubleshooting information, check reference link 3.

Reference links:
https://docs.aws.amazon.com/athena/latest/ug/other-notable-limitations.html
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-common.html
https://docs.aws.amazon.com/glue/latest/dg/troubleshooting-contact-support.html
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions