download events from EventBridge into a Docker image for automated testing?

Question

What's the most streamlined way to download events from EventBridge into a docker image to drive automated testing e.g. via LocalStack or other custom testing/validation processes?

I'd like past events to get baked into an image to avoid massive reprocessing with every incremental test. There will be other tests which need to reprocess the entire stream, but that's somewhat of a different problem.

The best idea I could come up with is to do all this within a Dockerfile 'RUN' statement: create a temporary SQS target, fire up an SQS downloader, fire up a replay, then poll the replay for completion, then poll the queue for completion. This seems a bit clunky and creates temporary infrastructure that requires a cleanup process.

Is there a better, simpler way?

SHAJAM · Accepted Answer

You can try a different approach. Using EventBridge send all events to Kinesis data streams.  The event pattern will be 
```
{   "source": [""] }
```
For target, choose Kinesis data stream and choose PutRecord for the operation. Then, enable  the rule

You need to update the IAM role as well and give access to Kinesis.

```
{
  "Effect": "Allow",
  "Action":  "kinesis:PutRecord*",
  "Resource": "arn:aws:kinesis:{AWS::Region}:{AWS::Account-id}:stream/my-kinesis-stream"
}
```

Then, from your docker, you can consume the data from Kinesis.

Why is this better? If your docker is down, you will miss out on the events when polling directly from EventBridge. However, with Kinesis, you can still poll previous events.

Kidd Ip · Answer

Below some comment:

1. Use LocalStack for EventBridge Emulation
* LocalStack is a great tool for emulating AWS services locally, including EventBridge. You can:
 
* Export your EventBridge events to a file (e.g., JSON format).
 
* Use LocalStack to replay these events during testing.
 
* Bake the exported events into your Docker image by copying the JSON file into the image and configuring LocalStack to load them at runtime.

```
FROM localstack/localstack
COPY events.json /etc/localstack/events.json
CMD ["localstack", "start"]

```
You can configure LocalStack to load the events from events.json during startup.

2. Export Events to S3 and Use a Pre-Baked File
* Export your EventBridge events to an S3 bucket using an AWS Lambda function or EventBridge rule.

* Download the events as a JSON file and bake it into your Docker image.

* During testing, load the events from the baked file into your testing framework.

```
FROM python:3.9
COPY events.json /app/events.json
WORKDIR /app
CMD ["python", "test_runner.py"]

```
Your test_runner.py script can load the events from events.json and simulate them.

3. Use AWS CLI for Replay
If you want to avoid temporary infrastructure, you can use the AWS CLI to fetch and replay events directly in your Dockerfile:

```
RUN aws events list-events --event-bus-name my-event-bus > /app/events.json

```
This approach avoids creating temporary SQS targets but requires AWS CLI credentials during the build process.

4. Pre-Process Events Outside Docker
* Instead of handling everything in the Dockerfile, pre-process the events outside Docker:
 
* Use a script to fetch and filter EventBridge events.
 
* Save the processed events as a file.
 
* Include the file in your Docker image.

This keeps your Dockerfile simpler and avoids the need for temporary infrastructure.

5. Use EventBridge Archive and Replay
* If you need to replay past events, consider using EventBridge's Archive and Replay feature. You can archive events and replay them during testing. While this requires AWS infrastructure, it simplifies the process of fetching and replaying events.

download events from EventBridge into a Docker image for automated testing?

Relevant content