How do I create and troubleshoot an Amazon EventBridge rule that triggers on S3 objects or operations?

Short description

A common use case requires Amazon EventBridge to trigger a workflow when it performs specific operations on an Amazon S3 buckets or objects. Amazon EventBridge offers various targets including AWS Lambda, Amazon Elastic Container Service (Amazon ECS) Task, API Destination, and Amazon Kinesis Data Firehose delivery stream. Use Amazon EventBridge to craft a rule or event pattern to run custom actions for specific Amazon S3 events.

Resolution

Choose one of the two options for setting up an Amazon EventBridge rule:

• Capture Amazon S3 events using AWS CloudTrail
• Capture S3 events directly from Amazon S3

Each option has its own set of metadata. Events offered by AWS CloudTrail include details from AWS API, AWS Identity and Access Management (IAM), and UserAgent. By contrast,  events sent from S3 include object details (for example, size).

Capture Amazon S3 events through AWS CloudTrail

AWS CloudTrail send S3 events to Amazon EventBridge when you create or update a trail to capture data events on S3 buckets. Here's a summary of the steps:

1. Create a CloudTrail trail to log events on the relevant S3 buckets.

2. Create a resource to process the S3 events. This can be any supported EventBridge target (for example, Lambda, ECS Task, API Destination, or Kinesis Firehose delivery stream)

3. Create an EventBridge rule with an event pattern to match the desired S3 events. The following example shows an event pattern that matches PutObject, DeleteObject, and DeleteObjects API calls made against a specific S3 bucket, with keys starting with feature1/ and feature2/:

   {
     "source": [
       "aws.s3"
     ],
     "detail-type": [
       "AWS API Call via CloudTrail"
     ],
     "detail": {
       "eventName": [
         "PutObject",
         "DeleteObject",
         "DeleteObjects"
       ],
       "requestParameters": {
         "bucketName": [
           "123456789012-prod-app"
         ],
         "key": [
           {
             "prefix": "feature1/"
           },
           {
             "prefix": "feature2/"
           }
         ]
       }
     }
   }

4. Associate the resource created earlier in this section with the EventBridge rule as a target.

For more information about how to capture data events, see Tutorial: Log Amazon S3 object-level operations using EventBridge.

Capture Amazon S3 events using AWS S3 service

This option doesn't require AWS CloudTrail. Here, the S3 bucket is configured to send all operations on that resource to the Amazon EventBridge service.

Activate S3 events to Amazon EventBridge

1. Sign in to the AWS Management Console, and then open the Amazon S3 console.
2. In the Buckets list, choose the name of the bucket that you want to activate events for.
3. Choose Properties.
4. Navigate to the Event Notifications section and find the Amazon EventBridge subsection. Choose Edit.
5. For Send notifications to Amazon EventBridge for all events in this bucket, choose On.

Create a resource to process the S3 events

This can be any supported EventBridge target (for example, Lambda, ECS Task, API Destination, Firehose delivery stream).

Create an Amazon EventBridge rule with an event pattern to match the S3 events

1. Create an EventBridge rule with an event pattern to match the S3 events. Here's an example of an event pattern that triggers on an object that's created or deleted within a specific S3 bucket. The pattern is a filter on filename parameters such as a size larger than 15 MB and with an extension of zip.

   {
     "source": [
       "aws.s3"
     ],
     "detail-type": [
       "Object Created",
       "Object Deleted"
     ],
     "detail": {
       "bucket": {
         "name": [
           "123456789012-prod-app"
         ]
       },
       "object": {
         "size": [
           {
             "numeric": [
               ">=",
               15728640
             ]
           }
         ],
         "key": [
           {
             "suffix": ".zip"
           }
         ]
       }
     }
   }

Associate the resource with the rule

Associate the resource with the rule created earlier, with the EventBridge rule as a target.

Best practices

Use the following best practices to resolve errors:

• If you use the CloudTrail solution, make sure that there's a trail for S3 buckets.

• For the S3 solution, make sure that the EventBridge events are activated on the desired bucket.

• Leverage EventBridge TriggeredRules and Invocations metrics to confirm successful matches to the rule and successful invocations of the targets:

• TriggeredRules data points indicate that the rule matched the incoming event. If there are no data points, then re-evaluate the event pattern. The event pattern is case-sensitive so "FAILED" and "failed" are processed differently. The Sandbox tool offers various example events from AWS Services and EventBridge partners. The tool validates incoming events against the event pattern.

• FailedInvocations metric indicate there's a problem with how EventBridge forwards events to the target. This might be related to permissions (AWS Identity and Access Management or resource policy) or target configuration.

• To match an event, all the fields in an event pattern must match. To troubleshoot the event pattern, remove fields to create a more generic event pattern. For more information, see How can I troubleshoot issues with Amazon EventBridge rules?

• EventBridge rules can have up to five targets. Temporarily attach a CloudWatch Logs group target or an Amazon Simple Notification Service (Amazon SNS) topic to troubleshoot problems. Both the log group and notifications capture the event forwarded to the target and help isolate misconfigurations. For more information, see How can I troubleshoot issues with Amazon EventBridge rules?

Related information

How can I create an Amazon EventBridge rule to automate responses to CloudTrail API calls?

Use Amazon S3 Event Notifications with Amazon EventBridge

Using dynamic Amazon S3 event handling with Amazon EventBridge

Tutorial: Log Amazon S3 object-level operations using EventBridge