Glue Job Fails | s3 to db/mysql | An error occurred while calling o87.getCatalogSink. None.get

0

When created a simple job using AWS Glue Studio, after successfully creating of crawlers & testing the connections as well. Using the visual editor with source as s3/bucket & target as MySql/RDS - the job is failing with the error An error occurred while calling o87.getCatalogSink. None.get

when the same job/script is being used to read the file from s3/bucket1 & target is being replaced as s3/bucket (instead of MySql) - the job is running fine.

PLEASE Assist in this.

1개 답변
1

Given the error message contains CatalogSink, It seems like the connection to MySQL is failing. There may be some portion that points to the exact error codes. To identify the exact error messages, searching through the ERROR logs with GlueExceptionAnalysisListener typically would lead me to the exact line that errors and the Error message. In this example, you should be able to find a specific error such as:

  • Driver not found
  • user name or password incorrect
  • database not found
  • error in query (table not found, column not found, query malformed, etc)

If you need to troubleshoot, please refer to this link - https://aws.amazon.com/premiumsupport/knowledge-center/glue-test-connection-failed/. To give you some pointers:

  • Ensure you are using correct connection parameters: jdbc:mysql://***aws_connection_url***amazonaws.com:3306/database_name
  • Glue connection refers to the correct VPC and subnet (if you are using a VPC)

Also, it would be useful if your Glue version is compatible with the MySQL version you are connecting to. If you are using Glue V3.0, it covers until MySQL V8.0.23. If your version is not compatible, you would have to write your own code.

  1. Download the JDBC driver for MySQL version you are using and Upload to S3
  2. Edit the Glue Job properties and add S3 URI in Dependent jars path - e.g. s3://yourS3bucket/path/mysql-connector-java-8.0.21.jar
  3. Modify the Glue Job e.g. Python script
datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf
(frame = dropnullfields3, catalog_connection = “GLUE-CONNECTION-NAME”, 
connection_options = { 
"customJdbcDriverS3Path": "s3://yourS3Bucket/path/mysql-connector-java-8.0.21.jar", 
"customJdbcDriverClassName": "com.mysql.cj.jdbc.Driver", 
"user": “#DBuserName#”, "password": “DbUsersPassword”, 
"url":"jdbc:mysql://dbname.url.region.rds.amazonaws.com/schemaName, 
"connectionType": "mysql", "dbtable": “tablenameInDb”, 
"database": “schemaName”}, transformation_ctx = "datasink4")
profile pictureAWS
답변함 일 년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠