1 Answers
0
Sharing some of your code might help here.
That said it appears there is a inconsistency between the types of frames you are using to process your data. I see this kind of message when I am trying to manipulate a glue frame with spark functions or vice versa.
answered 13 days ago
Relevant questions
An error occurred while calling o110.pyWriteDynamicFrame. null
asked 19 days agoCopying data from sql server to snowflake with AWS GLUE
asked 2 months agoTrying to Run a Job in Glue with Snowflake. Job Failed
asked 5 months agoAWS Glue gives error: "An error occurred while calling o285.save. File already exists:"
asked 8 months agoGlue Job Error (An error occurred while calling o100.pyWriteDynamicFrame. Illegal empty schema)
Accepted Answerasked 5 months agoAn error occurred while calling o95.pyWriteDynamicFrame.
asked 6 months agoAn error occurred while calling o187.pyWriteDynamicFrame. Unable to execute HTTP request:
asked 3 months agoWhat's the meaning of an AWS Glue Database name in italic?
Accepted Answerasked 4 months agoGlue ETL PySpark Job Fails after Upgrade from Glue Version 2.0 to 3.0 error occurred while calling pyWriteDynamicFrame EOFException occurred while reading the port number from pyspark.daemon's stdout
asked 6 months agoAn error occurred while calling o280.pyWriteDynamicFrame
asked 4 years ago
Hey Tim, thank you so much for taking the time to help. Please see below all the code, I created the job through the visual interface so the code was automatically populated:
import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job
args = getResolvedOptions(sys.argv, ["JOB_NAME"]) sc = SparkContext() glueContext = GlueContext(sc) spark = glueContext.spark_session job = Job(glueContext) job.init(args["JOB_NAME"], args)
Script generated for node snowflake-test-connector
snowflaketestconnector_node1 = glueContext.create_dynamic_frame.from_options( connection_type="custom.jdbc", connection_options={ "query": "select * from PROD.SEMARCHY_FLATTEN.HUB_1_INSTITUTIONS_COUNTRIES;", "connectionName": "snowflake_hardcoded", }, transformation_ctx="snowflaketestconnector_node1", )
Script generated for node ApplyMapping
ApplyMapping_node2 = ApplyMapping.apply( frame=snowflaketestconnector_node1, mappings=[ ("name", "string", "name", "string"), ("countrycode", "string", "country_code", "string"), ("b_loadid", "string", "b_loadid", "string"), ("b_classname", "string", "b_classname", "string"), ], transformation_ctx="ApplyMapping_node2", )
Script generated for node PostgreSQL table
PostgreSQLtable_node3 = glueContext.write_dynamic_frame.from_catalog( frame=Apply
Also when I selected the classic script option I got this error:
Error generating the script [gluestudio-service.eu-west-2.amazonaws.com] createScript: InvalidInputException: Error parsing node in DAG: {"previewAmount":0,"dataPreview":false,"mapping":[{"fromPath":["name"],"toKey":"name","toType":"string","fromType":"string","children":null,"dropped":false},{"fromPath":["countrycode"],"toKey":"country_code","toType":"string","fromType":"string","children":null,"dropped":false},{"fromPath":["b_loadid"],"toKey":"b_loadid","toType":"string","fromType":"string","children":null,"dropped":false},{"fromPath":["b_classname"],"toKey":"b_classname","toType":"string","fromType":"string","children":null,"dropped":false}],"codeGenVersion":1,"inputs":["node-1"],"name":"ApplyMapping","classification":"Transform","type":"ApplyMapping","parentsValid":true,"calculatedType":"","nodeId":"node-2","generatedNodeName":"ApplyMapping_node2"}