Glue: Get Sink- Error "unsupported operand type(s) for +: 'JavaMember' and 'str'"

1

Hello Members, Please help with below issue, I am not able to find in documentation

I am trying below :


silver_target = glueContext.getSink( path='s3://bucketpath', connection_type="s3", updateBehavior='UPDATE_IN_DATABASE', partitionKeys=["pyear"], enableUpdateCatalog=True, transformation_ctx="silver_target", )

silver_target.setCatalogInfo( catalogDatabase=gluecatalogdb, catalogTableName=tableName ) silver_target.setFormat("glueparquet") silver_target.writeFrame(newdf)


newdf is : dataframe for existing data + new incremental data so I tired : appended_df=existing_df.union(newincremental_df) newdf=appended_df.repartition(1)


After executing this I am getting error as TypeError: unsupported operand type(s) for +: 'JavaMember' and 'str'


I check the schema, count I see everything is matching , not sure what is going on in backend

Please help. Thanks

  • Can you please share the rest of the code ? This sounds like a problem in pure python code.

2 Answers
0
Accepted Answer

Hi @rePost-User-5744318,

Could you please try to convert the glue dynamic frame to spark data frame, and apply union transformation. It may give some better error logs, so that further troubleshooting can be performed. Thank you :)

answered 8 months ago
  • Once I converted the dynamic frame to DF, union worked as expected, thank you

  • I am facing similar issue, you can read any two dynamicFrames and apply union and then do a map/filter it would give you an error that javamember' object has no attribute '_get_object_id'.

    Converting to DataFrame and doing union works but that is not a solution for those who do not want the conversion overhead.

0

That's not a really a data error, that's sounds like API usage error or some print statement that is trying to concatenate.
The stacktrace should tell you the line/API that is not used correctly

profile pictureAWS
EXPERT
answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions