How can I run a post action script while writing to redshift from aws glue?

Question

```
        df.toDF().write.format("jdbc").\
        option("url", "").\
        option("dbtable", f"public.{tableName}_staging").\
        option("user", "").\
        option("password", "").\
        option("postactions", f"DROP TABLE IF EXISTS $staging").\
        mode('append').save()
```

I have mention the post actions script in the option but it is still not executed, is there a different way to execute the post action script from redshift in aws glue notebook
I used the script from https://stackoverflow.com/questions/49735489/upsert-from-aws-glue-to-amazon-redshift

Gonzalo Herreros · Answer

Please note postaction is a feature of Glue DynamicFrame while you are converting there to a standard Spark DataFrame (toDF()) and using the standard JDBC driver instead of the Glue native Redshift connector.

My advice is that you write that DynamicFrame using a sink provided by GlueContext (like in the example you reference). Otherwise you would have to open your own connector to Redshift to run that DROP DDL (e.g. opening a JDBC connection or a Python library like psycopg2)

How can I run a post action script while writing to redshift from aws glue?

Add your answer

Relevant content