How can I troubleshoot issues with Amazon EventBridge rules?

Short description

Use this validation workflow to isolate the problem:

1. Confirm that the relevant event occurred (for example, an AWS CloudTrail API call or a Simple Storage Service (Amazon S3) object upload).
2. Confirm that the event matched the rule event pattern.
3. Confirm that the target successfully processed the event (for example, the event invoked an AWS Lambda function).

Resolution

Use these steps to troubleshoot issues related to Amazon EventBridge rules:

Validate an incoming event

Validate the incoming event in one or more of these ways:

• If AWS CloudTrail captured the event, then analyze the CloudTrail logs. Verify that the API call occurred at the expected time and matches all the fields in event pattern.
  Note: Some AWS services are available only in the us-east-1 Region. For example, AWS Identity and Access Management (IAM) API calls are published only in the us-east-1 Region. If a rule to match IAM events was created in another Region, these events are not forwarded to EventBridge. As a result, the rule isn't triggered by these events.
• The Amazon EventBridge console provides the EventBridge tool Sandbox that contains example AWS and partner events. Use the tool to match the proposed event pattern against AWS, partner, and custom events. Or, use the TestEventPattern API to confirm a match between the event pattern and the event.
• EventBridge provides TriggeredRules metrics to report a rule for successfully matched events. The timestamps of the TriggeredRules datapoints show when the event occurred.

Review CloudWatch metrics

Follow these steps to review the CloudWatch metrics for an Amazon EventBridge rule:

1. Open the CloudWatch console.
2. Select All Metrics.
3. Select the AWS/Events namespace.
4. Select the TriggerRules, Invocations, and FailedInvocations (if available) metrics for the rule in question. You can view these metrics with the SUM statistic.

Validate the event pattern

To match an event, all the fields in an event pattern must match. Use the Sandbox tool to test the desired event against an event pattern with a few fields. For example, use a simple event pattern such as what's listed here:

{
    "source": ["aws.s3"]
}

Increase the event pattern's complexity if you get successful matches. Each field that you add to the event pattern filters the events that it matches and narrows its focus.

Below is an example event pattern. The pattern matches a list of Amazon S3 API calls against a specific bucket for an object that starts with either of two strings: feature1 and feature2.

{
  "source": [
    "aws.s3"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventName": [
      "PutObject",
      "DeleteObject",
      "DeleteObjects"
    ],
    "requestParameters": {
      "bucketName": [
        "123456789012-prod-app1"
      ],
      "key": [
        {
          "prefix": "feature1/"
        },
        {
          "prefix": "feature2/"
        }
      ]
    }
  }
}

Add fields gradually to isolate the fields that don't match the desired events.

Validate the target

• The Invocations metric publishes a datapoint when a rule forwards an event to a target. This represents successful and failed invocations.

• Amazon EventBridge issues a FailedInvocations datapoint when it permanently fails to invoke the target. FailedInvocations indicates problems with the target configuration or issues that arise from inadequate permissions.

• Amazon EventBridge must be given the appropriate permissions to invoke the target. Depending on the target, the AWS Identity and Access Management (IAM) role or a resource policy grants permissions.

• When you create rules using the Amazon EventBridge console, the console automatically adds the required permissions to the related resources.
  Note: Explicitly configure the permissions if you deploy the rule using AWS SDK, the AWS Command Line Interface (AWS CLI), or AWS CloudFormation.

• If there aren't any FailedInvocations datapoints, then Amazon EventBridge delivered the event to the target successfully. However, the target might not be functioning correctly. For example, an AWS Lambda target might encounter errors while running or when throttling. To get more information, review the target's Amazon CloudWatch metrics and any relevant logs.

• Associate an Amazon Simple Queue Service (Amazon SQS) dead-letter queue (DLQ) with the target. Events that weren't delivered to the target are sent to the dead-letter queue. You can use this method to get greater details about failed events. Review the following snippet of a message retrieved from the DLQ for a failed event:

  "MessageAttributes": {
                  "ERROR_CODE": {
                      "StringValue": "NO_PERMISSIONS",
                      "DataType": "String"
                  },
                  "ERROR_MESSAGE": {
                      "StringValue": "User: events.amazonaws.com is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-east-1:123456789012:function:Hello_World because no resource-based policy allows the lambda:InvokeFunction action (Service: AWSLambdaInternal; Status Code: 403; 
  Error Code: AccessDeniedException; Request ID: 6635c4c4-9c54-416e-bc40-ef25bad0aca5; Proxy: null)",
                      "DataType": "String"
                  },

  The message shows that Amazon EventBridge service doesn't have the required permissions to invoke the Lambda target.

• Amazon EventBridge rules support up to five targets. Temporarily, attach an Amazon CloudWatch Logs group target or an Amazon Simple Notification Service (Amazon SNS) topic. This action captures the events forwarded to the targets or isolates target misconfigurations. The log or topic provides these services:

• Confirms the event body and specific fields that must be parsed and processed by the target

• Confirms that the event was delivered to the target

• Confirms that the input transformer was correctly configured

Related information

Troubleshooting Amazon EventBridge

Why wasn't my Lambda function triggered by my EventBridge rule?

How do I troubleshoot errors in the Amazon EventBridge Scheduler?