Send customised notification of Glue job using SNS

Question

I need to send notification having ** job execution time**, **number of records processed** and the rest of the things using SNS. Below is the template I am using, but I need these two inputs too in my notification:
{
  "detail-type": ["Glue Job State Change"],
  "source": ["aws.glue"],
  "detail": {
    "jobName": ["job_name1","job_name2"],
    "severity": ["ERROR"],
    "state": ["FAILED", "ERROR", "TIMEOUT", "STOPPED"]
  }
} 
Can you please help me.

Answer

Hi ,

the 2 additional inputs (job execution time, number of records processed) you require are not generated automatically by Glue.

The job execution time you can retrieve it using the [Glue APIs **get_job_run**](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get_job_run) passing the latest run_id.

The number of record preprocessed you would need to calculate it in the job and then export it in and external file or db table you can query to retrieve it and add it to the SNS message before you send it.

hope this helps.

Send customised notification of Glue job using SNS

Relevanter Inhalt