Glue: Get Sink- Error "unsupported operand type(s) for +: 'JavaMember' and 'str'"

1

Hello Members, Please help with below issue, I am not able to find in documentation

I am trying below :


silver_target = glueContext.getSink( path='s3://bucketpath', connection_type="s3", updateBehavior='UPDATE_IN_DATABASE', partitionKeys=["pyear"], enableUpdateCatalog=True, transformation_ctx="silver_target", )

silver_target.setCatalogInfo( catalogDatabase=gluecatalogdb, catalogTableName=tableName ) silver_target.setFormat("glueparquet") silver_target.writeFrame(newdf)


newdf is : dataframe for existing data + new incremental data so I tired : appended_df=existing_df.union(newincremental_df) newdf=appended_df.repartition(1)


After executing this I am getting error as TypeError: unsupported operand type(s) for +: 'JavaMember' and 'str'


I check the schema, count I see everything is matching , not sure what is going on in backend

Please help. Thanks

  • Can you please share the rest of the code ? This sounds like a problem in pure python code.

2 Respuestas
0
Respuesta aceptada

Hi @rePost-User-5744318,

Could you please try to convert the glue dynamic frame to spark data frame, and apply union transformation. It may give some better error logs, so that further troubleshooting can be performed. Thank you :)

respondido hace 9 meses
  • Once I converted the dynamic frame to DF, union worked as expected, thank you

  • I am facing similar issue, you can read any two dynamicFrames and apply union and then do a map/filter it would give you an error that javamember' object has no attribute '_get_object_id'.

    Converting to DataFrame and doing union works but that is not a solution for those who do not want the conversion overhead.

0

That's not a really a data error, that's sounds like API usage error or some print statement that is trying to concatenate.
The stacktrace should tell you the line/API that is not used correctly

profile pictureAWS
EXPERTO
respondido hace 9 meses

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas