Glue Job Fails | s3 to db/mysql | An error occurred while calling o87.getCatalogSink. None.get

0

When created a simple job using AWS Glue Studio, after successfully creating of crawlers & testing the connections as well. Using the visual editor with source as s3/bucket & target as MySql/RDS - the job is failing with the error An error occurred while calling o87.getCatalogSink. None.get

when the same job/script is being used to read the file from s3/bucket1 & target is being replaced as s3/bucket (instead of MySql) - the job is running fine.

PLEASE Assist in this.

1 Answer
1

Given the error message contains CatalogSink, It seems like the connection to MySQL is failing. There may be some portion that points to the exact error codes. To identify the exact error messages, searching through the ERROR logs with GlueExceptionAnalysisListener typically would lead me to the exact line that errors and the Error message. In this example, you should be able to find a specific error such as:

  • Driver not found
  • user name or password incorrect
  • database not found
  • error in query (table not found, column not found, query malformed, etc)

If you need to troubleshoot, please refer to this link - https://aws.amazon.com/premiumsupport/knowledge-center/glue-test-connection-failed/. To give you some pointers:

  • Ensure you are using correct connection parameters: jdbc:mysql://***aws_connection_url***amazonaws.com:3306/database_name
  • Glue connection refers to the correct VPC and subnet (if you are using a VPC)

Also, it would be useful if your Glue version is compatible with the MySQL version you are connecting to. If you are using Glue V3.0, it covers until MySQL V8.0.23. If your version is not compatible, you would have to write your own code.

  1. Download the JDBC driver for MySQL version you are using and Upload to S3
  2. Edit the Glue Job properties and add S3 URI in Dependent jars path - e.g. s3://yourS3bucket/path/mysql-connector-java-8.0.21.jar
  3. Modify the Glue Job e.g. Python script
datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf
(frame = dropnullfields3, catalog_connection = “GLUE-CONNECTION-NAME”, 
connection_options = { 
"customJdbcDriverS3Path": "s3://yourS3Bucket/path/mysql-connector-java-8.0.21.jar", 
"customJdbcDriverClassName": "com.mysql.cj.jdbc.Driver", 
"user": “#DBuserName#”, "password": “DbUsersPassword”, 
"url":"jdbc:mysql://dbname.url.region.rds.amazonaws.com/schemaName, 
"connectionType": "mysql", "dbtable": “tablenameInDb”, 
"database": “schemaName”}, transformation_ctx = "datasink4")
profile picture
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant questions