How do I troubleshoot issues with EventBridge rules?

Resolution

Note: The following troubleshooting steps apply to all EventBridge target types. If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Check your configuration

To make sure that you correctly configured your EventBridge rules, validate the following settings.

• The rule is set to Enable, not Disable.
• Your event pattern uses valid JSON syntax.
• The EventBridge role for your target has the required AWS Identity and Access Management (IAM) permissions.
• The target exists and is in the same AWS Region as the EventBridge rule.
  Note: To configure cross-Region events, see Sending and receiving events between AWS Regions in Amazon EventBridge.
• The resource-based policy allows EventBridge to invoke the target.
• The event source publishes events to the correct Region.

Confirm that the event occurred

Note: AWS global services are available only in the us-east-1 Region. If you created a rule for these services in another Region, then the services don't forward the events to EventBridge. As a result, EventBridge doesn't run the rule.

Check the CloudWatch metrics

To identify the cause of the issue, check the Amazon CloudWatch console in the AWS/Events namespace for the following EventBridge metrics:

• Check TriggeredRules to check whether EventBridge ran the rule. If the value is 0, then check your API calls to validate that the event occurred. You can use the timestamp of this metric to check when the event occurred.
• Check Invocations to check whether EventBridge invoked the target. If there's a value for TriggeredRules but not for Invocations, then the rule didn't invoke the target. To resolve this issue, check the rule's target configuration.
• Check FailedInvocations to check whether the target invocation failed. If there's a value for Invocations and also for FailedInvocations, then the issue occurred in the target.

Check your configuration's API calls

Check the AWS CloudTrail Event history to verify that the event API call occurred at the expected time. Also, make sure that the event matches all the fields in the event pattern.

Confirm that the event matched the rule event pattern

For EventBridge to run a rule, all the fields in an event pattern must match the event for that rule. Use the EventBridge Sandbox tool to test whether the event matches a specific event pattern field. It's a best practice to start tests with an event pattern that uses few fields.

Example event pattern:

{    "source": ["aws.s3"]}

If you get a successful match for that event pattern, then increase the event pattern's complexity. Each new field narrows the focus of the event pattern test.

If you don't know your event scheme, then temporarily attach an Amazon CloudWatch Logs group target or an Amazon Simple Notification Service (Amazon SNS) topic. Review the event structure that EventBridge delivers to the CloudWatch Logs group target or Amazon SNS topic. Then, rewrite the pattern to match the event.

Example event pattern:

{  "source": [    "aws.s3"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventName": [
      "PutObject",
      "DeleteObject",
      "DeleteObjects"
    ],
    "requestParameters": {
      "bucketName": [
        "123456789012-prod-app1"
      ],
      "key": [
        {
          "prefix": "feature1/"
        },
        {
          "prefix": "feature2/"
        }
      ]
    }
  }
}

Note: The preceding pattern is for a list of Amazon Simple Storage Service (Amazon S3) API calls. It matches the API calls against a specific Amazon S3 bucket for an object that starts with either feature1 or feature2.

You can also use the TestEventPattern API to check whether the event matches the event pattern.

Use the following common event patterns based on the AWS service that has the event.

Amazon Elastic Compute Cloud (Amazon EC2) instance state changes:

{
    "source": ["aws.ec2"],
    "detail-type": ["EC2 Instance State-change Notification"],
    "detail": {
        "state": ["running", "stopped"]
    }
}

Amazon Relational Database Service (Amazon RDS) failovers or failures:

{
    "source": ["aws.rds"],
    "detail-type": ["RDS DB Instance Event"],
    "detail": {
        "EventCategories": ["failover", "failure"]
    }
}

AWS Step Functions executions that fail, time out, or are cancelled:

{
  "source": ["aws.states"],
  "detail-type": ["Step Functions Execution Status Change"],
  "detail": {
    "status": ["FAILED", "TIMED_OUT", "ABORTED"],
    "stateMachineArn": ["arn:aws:states:us-east-1:123456789012:stateMachine:MyStateMachine"]
  }
}

Check the event pattern syntax

Make sure that your event pattern adheres to the following syntax:

• Use quotes around all field names or values.
• Use array notation, not equals. For example, use "state": ["running"] not "state": "running".
• Validate that the nesting levels in your detail object are correct for your configuration.
• Make sure that all fields match. Event patterns use AND logic, not OR.
• Make sure that you correctly use wildcards based on the event schema.

Confirm that the target successfully processed the event

Troubleshoot failed invocations

If there's a data point in FailedInvocations in the CloudWatch console, then check your target configuration and permissions for issues.

EventBridge must have permission to invoke a target. When you use EventBridge to create rules, the console automatically adds the required permissions to the related resources. However, if you use AWS SDKs, the AWS CLI, or AWS CloudFormation to deploy the rule, then you must configure permissions. You can use an IAM role that has the required EventBridge policies attached or a resource-based policy.

Configure a dead-letter queue

If an invocation fails, then EventBridge automatically retries it with exponential backoff.

To identify the cause for FailedInvocations, complete the following steps to associate an Amazon Simple Queue Service (Amazon SQS) dead-letter queue with the target:

1. Open the EventBridge console.
2. Choose Rules, and then choose your rule.
3. Under Targets, select the target, and then choose Edit.
4. Expand Additional settings, and then configure the Maximum age of event and Retry attempts values.
   Note: EventBridge sends the event to the dead-letter queue only after it exceeds the maximum event of the age and the number of retry attempts. By default, these values are set to the maximum 24 hours and 185 retry attempts.
5. Under Dead-letter queue, select an Amazon SQS queue in the same or different AWS account. Then, choose the SQS queue that you use to capture failed events after EventBridge completes its retry events.
   Note: If you don't have an existing queue, then create an SQS queue.
6. Choose Next or Update rule to save your changes.

Note: If there aren't any FailedInvocations datapoints, then EventBridge delivered the event to the target successfully and the issue is with the target. For example, an AWS Lambda target might encounter errors when it runs or throttles. To troubleshoot these issues, check the CloudWatch metrics and logs for the target.

Troubleshoot failed invocations with the dead-letter queue

If there are any failed invocations for the rule, then the dead-letter queue logs the error message.

To review the error, complete the following steps:

1. Open the Amazon SQS console.
2. Choose Queues, and then select your dead-letter queue.
3. Choose Send and receive messages.
4. Choose Poll for messages.
5. Select an error message from the list.
6. Under Attributes, check the error details.
   The following example message shows that EventBridge doesn't have the required permissions to invoke the Lambda target:

   "MessageAttributes": {                "ERROR_CODE": {
                       "StringValue": "NO_PERMISSIONS",
                       "DataType": "String"
                   },
                   "ERROR_MESSAGE": {
                       "StringValue": "User: events.amazonaws.com is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-east-1:123456789012:function:Hello_World because no resource-based policy allows the lambda:InvokeFunction action (Service: AWSLambdaInternal; Status Code: 403; 
   Error Code: AccessDeniedException; Request ID: 6635c4c4-9c54-416e-bc40-ef25bad0aca5; Proxy: null)",
                       "DataType": "String"
                   },

Related information

Troubleshooting Amazon EventBridge

Why didn't my EventBridge rule invoke my Lambda function?

How do I troubleshoot errors in EventBridge Scheduler?

Using dead-letter queues to process undelivered events in EventBridge