Glue Relationalize recursion / depth
0
Does relationalize only go one layer in depth? I have json objects that have 2 to 3 layers of struct arrays and I get this error on anything past the first array layer : An error occurred while calling o136.pyWriteDynamicFrame. Bad dbtable value expecting either schema.table or table
what am I missing here? code example (works as expected with objects that have 1 layer of array depth)
PostgreSQLtable_node3 = glueContext.create_dynamic_frame.from_catalog(database="tekmetric",table_name="repair_orders")
resolvechoice2 = ResolveChoice.apply(frame=PostgreSQLtable_node3, choice="make_struct", transformation_ctx="resolvechoice2")
dropnullfields3 = DropNullFields.apply(frame=resolvechoice2, transformation_ctx="datasource0").drop_fields("jobs.parts.dotNumbers")
dfc = Relationalize.apply(frame=dropnullfields3, staging_path="s3://tekmetric-temp/repair-orders/", name="repair_orders", transformation_ctx="relationalize4")
for df_name in dfc.keys():
m_df = dfc.select(df_name)
if m_df.count():
glueContext.write_dynamic_frame.from_jdbc_conf(frame = m_df, catalog_connection = "postgres", connection_options = {"dbtable": df_name, "database": "tekmetric"})
asked 24 days ago15 views
1 Answers
0
Fixed this myself - it wasn't an issue with Relationalize per se ... writing to Postgres, the write_dynamic_frame did not enjoy periods in the table names. Fix :
for df_name in dfc.keys():
m_df = dfc.select(df_name)
final = df_name.replace('.val.','_')
if m_df.count():
glueContext.write_dynamic_frame.from_jdbc_conf(frame = m_df, catalog_connection = "postgres", connection_options = {"dbtable": final, "database": "tekmetric"})
answered 24 days ago
Relevant questions
Best way to overcome HIVE_PARTITION_SCHEMA_MISMATCH error in Athena while preserving structure of structs?
asked 3 months agoQuestions for querying from spectrum data stored STRUCT-STRUCT-ARRAY data type.
asked 2 months agoXML interpret one struct as an array
Accepted Answerasked 7 months agoBoto 3 textract Unkown Service
Accepted Answerasked 3 years agoCTAS can't handle empty arrays or maps
asked 4 years agoGlue Relationalize recursion / depth
asked 24 days agoGlue ETL Job not working with error: o122.relationalize. com.amazonaws.services.glue.util.HadoopDataSourceJobBookmarkState cannot be cast to com.amazonaws.services.glue.util.RelationalizeJobBookmarkSt
asked 6 months agoAWS Glue Dynamid Dataframe relationalize
Accepted Answerasked 3 years agoCast struct column to array
asked 3 months agoGlue job error : run ID: jr_f96799827354866ac2e798fb8b40d5781284e5ed5b3a4ff
asked a year ago