AWS Glue database JDBC writes PARTIAL data.. how to avoid that ?


How to make sure a particular dynamic frame is not committed partially to MySQL database ? Is there an option in write_from_catalog to achieve that ?

asked 10 months ago244 views
1 Answer

Not really, you would need a distributed transactions coordination between nodes that neither Glue nor Spark support.
What you can do is instead of writing to the final table, write to a temporary table and once the write operation is complete issue a DB command (using JDBC or other library) to do the merge inside the DB in a transactional way.

profile pictureAWS
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions