pyspark.sql.utils.AnalysisException• Cannot resolve column name "previous_project_id among ()
Hi, I'm relatively new to AWS glue and was having trouble in the following transformation codes:
DataSource4 = glueContext.create_dynamic_frame.from_catalog(database = "beta", table_name = "[table_name]", transformation_ctx = "DataSource4")
Transform9 = ApplyMapping.apply(frame = DataSource4, mappings = [("project_unique_id", "int", "(src) project_unique_id", "int")], transformation_ctx = "Transform9")
Transform9DF = Transform9.toDF()
Transform3DF = Transform3.toDF()
Transform12 = DynamicFrame.fromDF(Transform3DF.join(Transform9DF, (Transform3DF['project_unique_id'] == Transform9DF['previous_project_id']), "leftanti"), glueContext, "Transform12")
The job is failing with error : raise AnalysisException:(s.split(': ',1)[1], stackTrace) 'Cannot resolve column name "previous_project_id" among ((src) project_unique_id);' on checking the tables, both columns "project_unique_id" and "previous_project_id" are filled with NULL values, could that be the reason for the above error?
Hi ,
could you please clarify the statement:
on checking the tables, both columns "project_unique_id" and "previous_project_id" are filled with NULL values
which table? the source table? or which DataFrame? Transform9DF or Transform3DF ?
could you post the schema of these 2 dataframes?
I might be mistaken, but I think that the apply mapping you are using is dropping any field other than "(src) project_unique_id" so when you trying to join on Transform9DF['previous_project_id'] this field is not found.
thank you
Relevant questions
New GlueJob UI: Cannot access my Glue Job
asked 5 months agopyspark.sql.utils.AnalysisException• Cannot resolve column name "previous_project_id among ()
asked 4 months agoAWS glue Upsert operation for few columns
asked 3 months agowhile importing s3 data into quicksight from glue database getting the following error
Accepted Answerasked 3 months agoHow to use transformation rule in DMS task to perform replace operation(remove space) for all columns
asked a month agoAWS Glue visual job
asked 2 months agoAWS DMS - Migration fails when column in column filter is in camel case
asked 2 months agoCast struct column to array
asked 3 months agoAWS Glue API get-partitions can't seem to cope with the partition column name "key" in the `expression` filter
asked 4 months agoException in User Class: org.postgresql.util.PSQLException : ERROR: column "id" does not exist
asked 6 months ago