Skip to content

How to pass EventBridge event data to a Glue Workflow

0

I have an EventBridge rule that triggers when a new file is added to an S3 bucket with the EventBridge target being a glue workflow. Now I want to pass event data from EventBridge to my glue workflow so that I can (for example) extract the name and path of the file that was added to S3 for further parsing. I've found a lot of documentation on passing event data from EventBridge to the target of a rule, and even how to transform that event data using input transformation, but I'm struggling to find any documentation on how I should format that event data so that my glue workflow picks up the event parameters that I want. No matter how I format the event data via input transformation, my glue job does not see any of the event parameters. Is this possible?

2 Answers
0

Hi,

I am assuming you already figured a way to obtain the event id which triggered the Glue workflow, if not this documentation will help.

The next step would be to fetch the entire event that corresponds to the event id of the workflow run. For this please refer to this post How to resolve a list of "eventIds" into event attributes?

Hope this is helpful!

Thanks, Rama

AWS
EXPERT
answered 2 years ago
0

I am also having the same issue. I don't not want to look up cloud trail by event id. I want to configure the eventbridge rule using input transformer and set the object key as argument as shown below: resource "aws_cloudwatch_event_rule" "glue_workflow_target" { "source": ["aws.s3"], "detail-type": ["Object Created"], "detail": { "bucket": { "name": ["some-bucket"] }, "object": { "key": [{ "wildcard": "some-prefix/*.extension" }] } } } resource "aws_cloudwatch_event_target" "glue_workflow_target" { rule = aws_cloudwatch_event_rule.s3_event_rule.name target_id = "glue-workflow" arn = aws_glue_workflow.example.arn role_arn = aws_event_bridge_glue_workflow..arn event_bus_name = "default" input_transformer { input_paths = { "objectKey" = "$.detail.object.key" } input_template = <<EOF { "--object_key": "<objectKey>" } EOF } } I have a glue trigger to pass this transformed input into glue workflow as follow resource "aws_glue_trigger" "glue_trigger"{ name = "example-trigger" type "EVENT" actions { "job_name" = "example-glue-job", "arguments" = { "--objectKey": "$.objectKey", } } ]'
workflow_name = "example-workflow" }

When I try and read the sys argv from Resolved Params following https://docs.aws.amazon.com/glue/latest/dg/workflow-run-properties-code.html I just see "--objectKey" , "$.objectKey" getting printed.

Can you show be directions on this. What is wrong with the above approch.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.