Hi, I'm relatively new to AWS glue and i'm having trouble with Glue ETL error.
What makes it strange is that this error is only on Dev env but not on Test env. Same code & configuration!
Also tried to .printSchema() but it doesn't show on the logs.
code:
JoinCheckinPartner_DF = JoinCheckinPartner_node1.toDF()
ApplyMappingCustomerAccessPass_DF = ApplyMappingCustomerAccessPass.toDF()
ApplyMappingAccessToken_DF = ApplyMappingAccessToken.toDF()
JoinCheckinPartnerCustomerAccessPass_DF = JoinCheckinPartner_DF.join(
ApplyMappingCustomerAccessPass_DF,
JoinCheckinPartner_DF.customer_access_pass_id == ApplyMappingCustomerAccessPass_DF.id_from_customeraccesspass_table,
how = 'left_outer',
)
Error:
23/12/13 15:46:36 ERROR ProcessLauncher: Error from Python:Traceback (most recent call last):
File "/tmp/02562bed0b28de087112b67ae97bf8681ae397a0d0083f36be9bc5f3c6b350a6.py", line 495, in <module>
JoinCheckinPartner_DF["customer_access_pass_id"] == ApplyMappingCustomerAccessPass_DF["id_from_customeraccesspass_table"]...
pyspark.sql.utils.AnalysisException: Cannot resolve column name "customer_access_pass_id" among ()
Glue Job Type: Spark
ETL Language: python 3
Glue Version: Glue 3.0
Thanks so much, Gonzalo. This explains it.