By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

get_connection timeout in AWG Glue job

0

I am following the articale below for doing update in MySQL using pymysql - https://awstip.com/aws-etl-glue-job-insert-update-support-7a396db832b. However, it looks like the job is timing out on below line -

connection = glue_client.get_connection(Name="<My Connection>")

I do not see any exception in the logs.

Test connection works fine.Also, the same connection worked when I used it in another job for insert-only created from Visual editor.

Topics
Analytics
Tags
AWS Glue
Language
English
rePost-User-0629761
asked 2 years ago215 views
1 Answer
1

To identify the exact error messages, searching through the ERROR logs with GlueExceptionAnalysisListener typically would lead me to the exact line that errors and the Error message. Please try that and let us know if you are still unable to identify the error

profile pictureAWS
Ananth Tirumanur
answered 2 years ago
  • rePost-User-0629761
    2 years ago

    Hi,

    I only see that the GlueExceptionAnalysisListener was registered in the error logs. No other exception is logged until the job times out.

    2022-10-03 21:44:07,711 INFO [Thread-12] spark.SparkContext (Logging.scala:logInfo(57)): Registered listener com.amazonaws.services.glueexceptionanalysis.GlueExceptionAnalysisListener.

    Also, note that I am using MySQL v8.

    I started fresh with a new job where the script was generated by Glue Studio. It ran fine without any issues. However, as soon as I added get_connection() call, the job started timing out.

  • Ananth Tirumanur
    2 years ago

    Please refer to Glue connection page where it mentions how to use the Glue Connection https://docs.aws.amazon.com/glue/latest/dg/connection-properties.html#connection-properties-jdbc

    Note that MySQL connections are possible for Amazon RDS instances only. If you are using other MySQL database, you dont have to use Glue Connections. You would need to upload the JDBC driver to S3. In the AWS Glue job console, Choose the AWS Glue job and edit the Job details tab, under Libraries > Referenced File Path and enter the S3 URI of JDBC file.

    If you are able to get the job to work without connections and the data looks right, you do not have to add a Glue connection.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content