EventBridge Pipes - Handle Enrichment Lambda Errors

Question

I have an EventBridge Pipe with DynamoDB streams source and Firehose delivery stream target. I also have a lambda function that performs enrichment on this data, and I have configured a DLQ to capture any errors.

In testing edge cases I found that when there is an unhandled exception in the enrichment lambda function, the pipe retries over and over, presumably until the source record expires after 24 hours. I would like to limit this in order to avoid repeatedly calling the lambda function in these scenarios. However, after enabling both `maximumRetryAttempts` and `maximumRecordAgeInSeconds` settings on the pipe, I do not see any change in behavior.

Based on the description on [this page in the EventBridge User Guide](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-error-troubleshooting.html), unhandled errors in the enrichment lambda function should be classified as a "Customer invocation error", which is described as having this error behavior: "For pipes with a stream source, EventBridge Pipes retries up to the maximum retry times configured on the pipe retry policy or until the maximum record age expires, whichever comes first." But I am not seeing this in practice.

Any guidance for capturing these kinds of enrichment errors while limiting the number of lambda retries?

re:Post Agent · Answer

The behavior you're experiencing with your EventBridge Pipe is indeed unexpected based on the documentation. However, there are a few things to consider and potential solutions to address your issue:

1. Verify Configuration: Ensure that you have correctly configured the `maximumRetryAttempts` and `maximumRecordAgeInSeconds` settings on your pipe. Double-check these values in the AWS Console or CLI to make sure they are set as intended.

2. Error Handling in Lambda: Instead of relying solely on the pipe's retry mechanism, implement proper error handling within your Lambda function. Catch exceptions and return an appropriate error response. This way, you can control how the pipe handles the error.

3. Partial Batch Response: For DynamoDB stream sources, you can implement partial batch processing. Configure your enrichment Lambda to return an object indicating which messages succeeded and which failed. This allows EventBridge to avoid reprocessing successfully processed messages in a failed batch.

4. Dead Letter Queue (DLQ): Since you mentioned you have configured a DLQ, ensure it's properly set up. For DynamoDB stream sources, you can configure a DLQ for the pipe to route events that fail processing after the specified number of attempts.

5. Synchronous Invocation: Make sure your Lambda function is invoked synchronously by the pipe. EventBridge Pipes needs to be aware of runtime errors to retry them properly.

6. CloudWatch Logs: Monitor your pipe's CloudWatch Logs for any error messages or unexpected behavior. This might provide insights into why the retry mechanism isn't working as expected.

7. AWS Support: If after verifying all the above, you still don't see the expected behavior, it might be worth contacting AWS Support. There could be a discrepancy between the documented behavior and the actual implementation.

Remember, EventBridge Pipes automatically retries on AWS failures, but for customer implementation failures (like unhandled exceptions in your Lambda), the pipe's polling throughput will gradually back off. In extreme cases of continuous errors, the pipe might even be automatically disabled.

By implementing these suggestions, particularly focusing on error handling within your Lambda function and utilizing partial batch responses, you should be able to better control the retry behavior and limit unnecessary Lambda invocations while still capturing enrichment errors effectively.  
**Sources**  
[Amazon EventBridge Pipes error handling and troubleshooting - Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-error-troubleshooting.html)  
[Amazon DynamoDB stream as a source for EventBridge Pipes - Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-dynamodb.html)

EventBridge Pipes - Handle Enrichment Lambda Errors

Add your answer

Relevant content