Glue Relationalize recursion / depth

0

Does relationalize only go one layer in depth? I have json objects that have 2 to 3 layers of struct arrays and I get this error on anything past the first array layer : An error occurred while calling o136.pyWriteDynamicFrame. Bad dbtable value expecting either schema.table or table

what am I missing here? code example (works as expected with objects that have 1 layer of array depth)

PostgreSQLtable_node3 = glueContext.create_dynamic_frame.from_catalog(database="tekmetric",table_name="repair_orders")
resolvechoice2 = ResolveChoice.apply(frame=PostgreSQLtable_node3, choice="make_struct", transformation_ctx="resolvechoice2")
dropnullfields3 = DropNullFields.apply(frame=resolvechoice2, transformation_ctx="datasource0").drop_fields("jobs.parts.dotNumbers")
dfc = Relationalize.apply(frame=dropnullfields3, staging_path="s3://tekmetric-temp/repair-orders/", name="repair_orders", transformation_ctx="relationalize4")

for df_name in dfc.keys():
    m_df = dfc.select(df_name)
    if m_df.count():
        glueContext.write_dynamic_frame.from_jdbc_conf(frame = m_df, catalog_connection = "postgres", connection_options = {"dbtable": df_name, "database": "tekmetric"})
asked 2 years ago886 views
1 Answer
1

Fixed this myself - it wasn't an issue with Relationalize per se ... writing to Postgres, the write_dynamic_frame did not enjoy periods in the table names. Fix :

for df_name in dfc.keys():
    m_df = dfc.select(df_name)
    final = df_name.replace('.val.','_')
    if m_df.count():
        glueContext.write_dynamic_frame.from_jdbc_conf(frame = m_df, catalog_connection = "postgres", connection_options = {"dbtable": final, "database": "tekmetric"})
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions