How to overwrite a bigquery table with AWS glue?

Question

This the code that I have right now: 
```
...
        # Define connection options for BigQuery.
        bigquery_options = {
            "connectionName": "Bigquery connection",
            "parentProject": "", 
            "writeMethod": "direct",
            "table": self.bigquery_table,
            "writeDisposition": "WRITE_TRUNCATE" # we always want to replace
        }
        self.glue_context.write_dynamic_frame.from_options(
            frame=final_dynamic_frame,
            connection_type="bigquery",
            connection_options=bigquery_options,
            transformation_ctx="GoogleBigQuery_Node"
        )
....
```

This doesn't seem to "WRITE_TRUNCATE", which is what I want. I can't find any articles about this either.

Brandon_P · Answer

Greetings,

Thank you for reaching out to AWS via this post.

From your post I understand that you need assistance with overwriting a BigQuery table via AWS Glue.

Please correct me if I have misunderstood anything.

To achieve this you may have to convert the DynamicFrame to a Spark DataFrame and then use the ` "mode": "overwrite"` parameter.

I hope you found the information provided helpful.

Wishing you a great day further!

How to overwrite a bigquery table with AWS glue?

Add your answer

Relevant content