fail a glue job if the called stored procedure fails

0

Hi all,

I'm using a glue job that reads CSV files from S3 and injects data to MySQL RDS.

at the end of my pyspark glue script, I call a stored procedure. the issue is the glue job status end by succeeding but the stored procedure execution fails.

is there any way to fail the glue job if the execution of the called stored procedure fails?

when glue script calls myStoredProcedure() it's an async call?

appreciate any help

import pymysql

with conn.cursor() as cur:
   insertQry="CALL myStoredProcedure()"
   cur.execute(insertQry)
   conn.commit()
   cur.close()
   conn.close()
   
job.commit()

************* OR **********************
from py4j.java_gateway import java_import
java_import(sc._gateway.jvm,"java.sql.Connection")
java_import(sc._gateway.jvm,"java.sql.DatabaseMetaData")
java_import(sc._gateway.jvm,"java.sql.DriverManager")
java_import(sc._gateway.jvm,"java.sql.SQLException")

source_jdbc_conf = glueContext.extract_jdbc_conf(glue_connection_name)
conn = sc._gateway.jvm.DriverManager.getConnection(source_jdbc_conf.get('url') + '/' + database_name, source_jdbc_conf.get('user'), source_jdbc_conf.get('password'))

 stmt = conn.createStatement();
 rs = stmt.executeQuery('call ' + stored_proc);
 conn.close()
1개 답변
1

Hello,

Stored procedure is a part of Python procedure and the stored procedure is not executed on glue resources. So, I do not believe it waits for confirmation whether it is successful or not

But, you can try to enclose the code for stored procedure within try/except block to capture the exception if the job fails. Glue job will capture the exception. Otherwise, there is no way Glue will know what is going on since stored procedure is a part of Python procedure. You can also try to set up sys.exit(1) to make sure it will throw something as well. Please refer bellow link for more details

REFERENCES:

https://stackoverflow.com/questions/6720119/setting-exit-code-in-python-when-an-exception-is-raised

AWS
지원 엔지니어
답변함 2년 전
AWS
전문가
검토됨 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠