Are EventBridge events (fully) passed to Glue Workflows?

2

Starting a Glue workflow with an EventBridge event set only (as Run properties) the IDs of the events that started the workflow. Are passing the full EventBridge events supported in Glue Workflows?

asked 2 years ago2426 views
2 Answers
0

I'm not sure I understand your question. Did you follow the instructions here?

https://docs.aws.amazon.com/glue/latest/dg/starting-workflow-eventbridge.html

profile pictureAWS
answered 2 years ago
  • I'm referring to access to the EventBridge event details from a Glue Job included in the workflow. According to the documentation, only the event Ids (and not the event details) are shared as Run Properties (aws:eventIds) among the jobs.

  • I have the same use case i.e. to pass the Eventbridge event details to a Glue workflow (as run properties). Currently there seems to be no way to achieve this. I also tried having lambda as a target for the Eventbridge rule and starting the Glue workflow from lambda. Even in that case, the glue client "start_workflow_run" API does not have an option to pass run properties

0

There is no native option to pass EventBridge event details to Glue job. But I used the following workaround to do this:

  1. You need to get an event ID from Glue workflow properties
event_id = glue_client.get_workflow_run_properties(Name=self.args['WORKFLOW_NAME'],
                       RunId=self.args['WORKFLOW_RUN_ID'])['RunProperties']['aws:eventIds'][1:-1]
  1. Get all NotifyEvent events for the last several minutes. It's up to you to decide how much time can pass between the workflow start and your job start.
response = event_client.lookup_events(LookupAttributes=[{'AttributeKey': 'EventName',
                                                         'AttributeValue': 'NotifyEvent'}],
                                                         StartTime=(datetime.datetime.now() - datetime.timedelta(minutes=5)),
                                                         EndTime=datetime.datetime.now())['Events']
  1. Check which event has an enclosed event with the event ID we get from Glue workflow.
for i in range(len(response)):
      event_payload = json.loads(response[i]['CloudTrailEvent'])['requestParameters']['eventPayload']
      if event_payload['eventId'] == event_id:
                event = json.loads(event_payload['eventBody'])

In the event variable you get the full content of the event that triggered Glue workflow.

profile picture
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions