1 Antwort
- Neueste
- Die meisten Stimmen
- Die meisten Kommentare
0
Sharing some of your code might help here.
That said it appears there is a inconsistency between the types of frames you are using to process your data. I see this kind of message when I am trying to manipulate a glue frame with spark functions or vice versa.
beantwortet vor 2 Jahren
Relevanter Inhalt
AWS OFFICIALAktualisiert vor einem Jahr- AWS OFFICIALAktualisiert vor 2 Jahren
- AWS OFFICIALAktualisiert vor 10 Monaten
Hey Tim, thank you so much for taking the time to help. Please see below all the code, I created the job through the visual interface so the code was automatically populated:
import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job
args = getResolvedOptions(sys.argv, ["JOB_NAME"]) sc = SparkContext() glueContext = GlueContext(sc) spark = glueContext.spark_session job = Job(glueContext) job.init(args["JOB_NAME"], args)
Script generated for node snowflake-test-connector
snowflaketestconnector_node1 = glueContext.create_dynamic_frame.from_options( connection_type="custom.jdbc", connection_options={ "query": "select * from PROD.SEMARCHY_FLATTEN.HUB_1_INSTITUTIONS_COUNTRIES;", "connectionName": "snowflake_hardcoded", }, transformation_ctx="snowflaketestconnector_node1", )
Script generated for node ApplyMapping
ApplyMapping_node2 = ApplyMapping.apply( frame=snowflaketestconnector_node1, mappings=[ ("name", "string", "name", "string"), ("countrycode", "string", "country_code", "string"), ("b_loadid", "string", "b_loadid", "string"), ("b_classname", "string", "b_classname", "string"), ], transformation_ctx="ApplyMapping_node2", )
Script generated for node PostgreSQL table
PostgreSQLtable_node3 = glueContext.write_dynamic_frame.from_catalog( frame=Apply
Also when I selected the classic script option I got this error:
Error generating the script [gluestudio-service.eu-west-2.amazonaws.com] createScript: InvalidInputException: Error parsing node in DAG: {"previewAmount":0,"dataPreview":false,"mapping":[{"fromPath":["name"],"toKey":"name","toType":"string","fromType":"string","children":null,"dropped":false},{"fromPath":["countrycode"],"toKey":"country_code","toType":"string","fromType":"string","children":null,"dropped":false},{"fromPath":["b_loadid"],"toKey":"b_loadid","toType":"string","fromType":"string","children":null,"dropped":false},{"fromPath":["b_classname"],"toKey":"b_classname","toType":"string","fromType":"string","children":null,"dropped":false}],"codeGenVersion":1,"inputs":["node-1"],"name":"ApplyMapping","classification":"Transform","type":"ApplyMapping","parentsValid":true,"calculatedType":"","nodeId":"node-2","generatedNodeName":"ApplyMapping_node2"}