AWS Glue job updating an existing table

Question

Using AWS Glue jobs, I want to insert, update and delete the rows in a Table in RDS Database based on the primary key. How do I achieve it.

Answer

This web site may help you.

https://medium.com/@thomaspt748/how-to-upsert-data-into-a-relational-database-using-apache-spark-part-1-python-version-b43b9761bbf2

I have created jobs to execute update, insert or delete sql statement on Oracle database on Amazon RDS by using JayDeBeApi, JPype or cx_oracle(if you use MySQL, then you may use mysql connector) libraries.

Answer

There are multiple ways one of them is -

1- Use Glue python shell job , copy the jdbc jar on S3 and refer then by initializing connection object and execute the SQL statement.
similar :- https://dev.mysql.com/doc/connector-python/en/connector-python-example-connecting.html

2- Create Mysql connection in AWS glue use connection object in Glue python shell job type in Glue studio or legacy ETL and follow step 1.

Other reference :- https://aws.amazon.com/blogs/big-data/building-aws-glue-spark-etl-jobs-by-bringing-your-own-jdbc-drivers-for-amazon-rds/

AWS Glue job updating an existing table

Relevant content