An error occurred while calling o110.pyWriteDynamicFrame. null

0

I have created a job in Glue where I am writing from Snowflake to a postgre database on AWS RDS.

I can preview the data in both connections but when I run the job it gives me this error:

An error occurred while calling o110.pyWriteDynamicFrame. null

asked 2 years ago790 views
1 Answer
0

Sharing some of your code might help here.

That said it appears there is a inconsistency between the types of frames you are using to process your data. I see this kind of message when I am trying to manipulate a glue frame with spark functions or vice versa.

profile pictureAWS
answered 2 years ago
  • Hey Tim, thank you so much for taking the time to help. Please see below all the code, I created the job through the visual interface so the code was automatically populated:

    import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job

    args = getResolvedOptions(sys.argv, ["JOB_NAME"]) sc = SparkContext() glueContext = GlueContext(sc) spark = glueContext.spark_session job = Job(glueContext) job.init(args["JOB_NAME"], args)

    Script generated for node snowflake-test-connector

    snowflaketestconnector_node1 = glueContext.create_dynamic_frame.from_options( connection_type="custom.jdbc", connection_options={ "query": "select * from PROD.SEMARCHY_FLATTEN.HUB_1_INSTITUTIONS_COUNTRIES;", "connectionName": "snowflake_hardcoded", }, transformation_ctx="snowflaketestconnector_node1", )

    Script generated for node ApplyMapping

    ApplyMapping_node2 = ApplyMapping.apply( frame=snowflaketestconnector_node1, mappings=[ ("name", "string", "name", "string"), ("countrycode", "string", "country_code", "string"), ("b_loadid", "string", "b_loadid", "string"), ("b_classname", "string", "b_classname", "string"), ], transformation_ctx="ApplyMapping_node2", )

    Script generated for node PostgreSQL table

    PostgreSQLtable_node3 = glueContext.write_dynamic_frame.from_catalog( frame=Apply

  • Also when I selected the classic script option I got this error:

    Error generating the script [gluestudio-service.eu-west-2.amazonaws.com] createScript: InvalidInputException: Error parsing node in DAG: {"previewAmount":0,"dataPreview":false,"mapping":[{"fromPath":["name"],"toKey":"name","toType":"string","fromType":"string","children":null,"dropped":false},{"fromPath":["countrycode"],"toKey":"country_code","toType":"string","fromType":"string","children":null,"dropped":false},{"fromPath":["b_loadid"],"toKey":"b_loadid","toType":"string","fromType":"string","children":null,"dropped":false},{"fromPath":["b_classname"],"toKey":"b_classname","toType":"string","fromType":"string","children":null,"dropped":false}],"codeGenVersion":1,"inputs":["node-1"],"name":"ApplyMapping","classification":"Transform","type":"ApplyMapping","parentsValid":true,"calculatedType":"","nodeId":"node-2","generatedNodeName":"ApplyMapping_node2"}

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions