How do I use a dead-letter queue to troubleshoot the FailedInvocations metric for EventBridge rules?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Create a standard Amazon SQS queue

Use the Amazon Simple Queue Service (Amazon SQS) console to create the SQS queue.

Complete the following steps:

1. Open the Amazon SQS console.
2. Choose Create queue.
3. For Type, choose Standard queue type.
4. Enter a name for your queue, for example myEventBridgeDLQ.
5. Choose Create queue.

Associate the SQS queue with the target of an EventBridge rule

Complete the following steps:

1. Open the EventBridge console.
2. In the navigation pane, choose Rules.
3. Select the EventBridge rule that failed to invoke your target, and then choose Edit.
4. In the navigation pane, choose Select target(s), and then choose Additional settings.
5. For Dead-letter queue, choose Select an Amazon SQS queue in the current AWS account to use as the dead-letter queue. Then, select your SQS queue in the Select an SQS queue dropdown list.
6. Choose Skip to Review and update.
7. Choose Update rule.

EventBridge must have permission to send events with failed invocations to the SQS queue. If you use the EventBridge console to associate a DLQ with the target of the EventBridge rule, then EventBridge automatically adds the permission. If you use AWS CLI, AWS SDK, or AWS CloudFormation, then you must manually create a resource-based policy that grants the required permission. You can run the set-queue-attributes command to modify the policy.

Important: If you encrypted the SQS queue, then you must create a customer managed key. You must also include the following permission in your AWS Key Management Service (AWS KMS) key policy:

{      "Sid": "Allow EventBridge to use the key",  
    "Effect": "Allow",  
    "Principal": {  
        "Service": "events.amazonaws.com"  
    },  
    "Action": [  
        "kms:Decrypt",  
        "kms:GenerateDataKey"  
    ],  
    "Resource": "*"  
}

For more information, see Configuring AWS KMS permissions.

Send events to your EventBridge event bus that match your EventBridge rule

To send events to your default event bus that match your event rule, change your AWS environment for AWS service events. If the EventBridge rule fails to invoke the target, then EventBridge sends all failed events to the DLQ.

To send events to your custom event bus that match your event rule, use the PutEvents API or the put-events command.

To check the FailedInvocations metric for your EventBridge rule, complete the following steps:

1. Open the EventBridge console.
2. Select your EventBridge rule, and then choose the Monitoring tab.
3. Look for MatchedEvents and TriggeredRules to confirm that the rule matched with an event.
4. Check Invocations and FailedInvocations to confirm that the rule tried to invoke the target.

Note: If the metrics don't have data points, then make sure that you correctly configured the rule pattern.

Retrieve the failed event that EventBridge sent to the DLQ

Complete the following steps:

1. Open the EventBridge console.
2. In the navigation pane, choose Rules.
3. Select the EventBridge rule that failed the invocation.
4. Choose the Targets tab, and then select the DLQ.
5. Choose Send and receive messages to open the Send and receive messages page on the SQS console.
6. Choose Poll for messages.
   Note: The Messages section displays a list of received messages. The list shows the message ID, date sent, size, and receive count for each message.
7. Select one of the messages. Then, choose View details to view the event that EventBridge failed to send to the target.
8. Choose the Attributes tab to view the ERROR_CODE, ERROR_MESSAGE, RULE_ARN, and TARGET_ARN attributes.
   Note: The ERROR_CODE and ERROR_MESSAGE attributes provide the reason for the event delivery failure.

Related information

How can I troubleshoot issues with Amazon EventBridge rules?

Amazon SQS permissions