How to specify a Kinesis partition key in the target of a EventBridge Pipe

Question

I'm wanting to use an EventBridge Pipe to pipe data from a various Kinesis streams into another Kinesis stream.

It looks like I can create a pipe that uses a Kinesis stream as a source, filter it to just the subset of data that I need, and pipe it to another Kinesis stream as the target.

The issue is that it seems like I have to specify a hard coded partition key for the target Kinesis stream. Am I not allowed to set a partition key path like the EventBridge Rule system? If I have to hard code the partition key, I wont be able to shard out, since the data will always go to only one shard.

Accepted Answer

Looks like this is supported. It's just not in the docs. It works like the partition key path from the event bridge rule.

If I set the partition key to `$.partitionKey`, then it would look at the incoming event and pull out the partitionKey from the source stream.

Answer

This can be accomplished by JSONPath .

Concretely, we will set a JSONPath for the partitionKey parameter (e.g $.data.customPK)

Then, when sending a payload to Pipes, on each payload they will add the partitionKey they want to shard to at the path customPk

E.g if they send { "customPK": "key1" },{ "customPK": "key2" },{ "customPK": "key3" } to the kinesis source, then the first event will be sent to partitionKey "key1", the second will be sent to "key2", and the third will be sent to "key3".

Please refer to the document : https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-input-transformation.html

How to specify a Kinesis partition key in the target of a EventBridge Pipe

Contenuto pertinente