Lake Formation Governed tables: truncate and reload?
Doing a POC with LF Governed table for a small dimension. I'd like to purge and reload the table each time it runs. I have tried to include a purge_table command within a governed table transaction but I receive the below error. I've tried adding a transactionId param to the purge_table command directly or within the options dict and neither has worked. Any ideas? Or anyone found a good way to truncate and reload a governed table?
"An error occurred while calling o119.purgeTable. Specify Either Transaction Id or Query AsOf Time"
Hello,
I tried to purge the table and reload but unable to reload the table. Could you please share the purge_table command you are using for us to test ? Also, we recommend you to please open a support ticket with us so that we can get on a screen-share and assist you by checking your Lake Formation configuration and parameters used.
Relevant questions
Master binary logs taking a lot of space and not being purged.
asked 3 years agoRecord Level Upsert on Governed Tables
asked a month agoAccess error : Spark query from AWS EMR with AWS Lake Formation
asked 5 months agoGoverned Tables not deleting smaller files after compaction
asked a month agoTable id found in `stl_load_errors` doesn't correspond to loaded table
Accepted Answerasked 3 years agoTRUNCATE DynamoDB table
Accepted Answerasked 2 days agoLake Formation Governed tables: truncate and reload?
asked 4 months agoIs it possible to specify DB snapshot in AWS Lake Formation?
Accepted Answerasked a year agoUpdating data in governed tables
asked 7 months agoAre you able to hide tables in a database using Lake Formation Tagging
asked 3 months ago
Thanks for your reply. See below. I'll also open a support ticket. Thank you
dest_path = <s3 path> db = <database name> tbl = <table name>
tx_id = glue_context.start_transaction(False)
sink = glue_context.getSink( connection_type="s3", path=dest_path, enableUpdateCatalog=True, transactionId=tx_id ) sink.setFormat("glueparquet") sink.setCatalogInfo( catalogDatabase=db, catalogTableName=tbl )
try: glue_context.purge_table(db, tbl, options={'transactionId':tx_id}) sink.writeFrame(glue_data_frame) glue_context.commit_transaction(tx_id) except Exception: glue_context.cancel_transaction(tx_id) raise job.commit()