EventBridge Pipes, running, but not being invoked by MSK Topic

Question

Hi,

I have a MSK running in a private subnet and have successfully setup up a Lambda and MSK trigger against a topic and also an EC2 as a consumer running in the same subnet.

I have setup an EventBridge Pipe using the same credentials (that the Lambda uses to auth to MSK) and the Pipe is in a "running" state with a target of CloudWatch using SASL auth.

The Topic has messages, both of the other consumers are triggered.

The EventBrdige Pipe is never invoked according to CloudWatch monitoring, existing messages on the topic and new ones added, no errors are reported.

Any ideas ?

Accepted Answer

It was permissions.... the Pipes auto created execution role and policy doesn't give Pipes the permissions required to use MSK as a source. I had augmented the auto created role with MSK Full Access but that does include below.

I noticed that my Lambda would only auth with SASL, so added below and then added same to the Pipes execution role.

Pipes showed as running, no errors reported, but clearly it couldn't connect or read. Wild that there are no errors and no logs that I could find. I'm guessing there must be a sequencing of setup issue.

https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html#msk-permissions-iam-policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:DescribeGroup",
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeClusterDynamicConfiguration"
            ],
            "Resource": "*"
        }
    ]
}

EventBridge Pipes, running, but not being invoked by MSK Topic

Relevant content