By using AWS re:Post, you agree to the Terms of Use
/Call a glue Job from within another without using vpc endppoints or SG/

Call a glue Job from within another without using vpc endppoints or SG

0

Hi all,

Is it possible to call a glue job/ or python script from within another glue job without passing by glue endpoint and adding a new rule in SG?

the following code uses glue endpoint and doesn't work without adding extra rules in the Security Group :

client = boto3.client('glue', 'AWS_REGION')
client.start_job_run(JobName='test_call_script')

I want to call an extra script that sends an email from within my original glue job without the need to use SES or glue VPC endpoints or opening new rules in the SG. just call another glue script that can send an email out of the box

1 Answers
1

Hi ,

if the first job has a connection and thus is linked to a VPC that will not be possible. The best way to do it , it is a full separation , exit from the initial job with an explicit exit code , and message or just writing the message ou want to send via email in a file on S3.

Then use a Glue Workflow or just a cloudwatch event to trigger a second job that read the output message from the first job and send the email. with the second job not attached to a VPC you do not need the VPC endpoints or SG rules.

Cloudwatch event for Glue Failed job:

{
    "version":"0",
    "id":"abcdef01-1234-5678-9abc-def012345678",
    "detail-type":"Glue Job State Change",
    "source":"aws.glue",
    "account":"123456789012",
    "time":"2017-09-07T06:02:03Z",
    "region":"us-west-2",
    "resources":[],
    "detail":{
        "jobName":"MyJob",
        "severity":"ERROR",
        "state":"FAILED",
        "jobRunId":"jr_0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
        "message":"JobName:MyJob and JobRunId:jr_0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef failed to execute with exception Role arn:aws:iam::123456789012:role/Glue_Role should be given assume role permissions for Glue Service."
    }
}

in the Glue script just use a python exit statement passing the message you want as exception and it should appear as meesage.

hope this is clear

EXPERT
answered 5 months ago
  • Thank you! suppose I want to use cloud watch event to trigger the second job that will send the email. how can I write an explicit code from the first job to the cloud watch event?

  • please check the revised answer.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions