How to see log of query made by Glue pyspark script against JDBC DB

Question

Is there a way to see or log the SQL query made by glue DynamicFrame calls against a JDBC database source?  We are trying to debug a long-running DynamicFrame Query and we are unable to see how the SQL call is being constructed.  Would like to know what SQL query statement is being sent to the DB server

Answer

Please try enabling the continuous logging feature for the glue job and see whether the required logs are visible or not. With the help of this feature, it is possible to view the real time logs in your Cloud watch console. Within these logs you can find executor logs, driver logs and spark job progress bar. I hope that these documentations might be of some help:
1. https://docs.aws.amazon.com/glue/latest/dg/monitor-continuous-logging.html
2. https://docs.aws.amazon.com/glue/latest/dg/monitor-profile-glue-job-cloudwatch-metrics.html

How to see log of query made by Glue pyspark script against JDBC DB

Contenuto pertinente