Lambda MSK trigger not working

It is very likely the security group attached to the Amazon MSK cluster is blocking the connection. Please create a self-referencing inbound rule on that security group on ports 9092 and 9094. Please check out a lab at https://amazonmsk-labs.workshop.aws/en/msklambda.html that provides step by step instructions.

There was already a self referencing inbound rule for All Traffic on the MSK security group. I also have the lambda security group added to the MSK security group. Not sure what to try next.

This error usually indicates a connectivity issue. Can you try spinning up an EC2 instance in one of the subnets as the Amazon MSK cluster, associate the same security group as Amazon MSK and try using one of the Kafka tools to access the Amazon MSK cluster? Please use the appropriate bootstrap broker string depending on whether TLS is enabled for clients on the cluster.
https://docs.aws.amazon.com/msk/latest/developerguide/create-client-machine.html, https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html, https://docs.aws.amazon.com/msk/latest/developerguide/produce-consume.html.

I have done that. I followed the guidance in the article I linked in my first post, which uses an EC2 kafka client to create a new topic and use the console producer/consumer scripts. This all works fine. I have the EC2 instance in it's own security group, but added that group to the MSK security group for all inbound traffic.

Hello, just to follow up I was able to resolve this by setting up the proper private subnets->public subnets->NAT Gateways in the vpc the MSK cluster was running in. Thank you for the assistance.

Hi,
Just want to know what happens if i have my Lambda in Public subnets and so is the MSK cluster. Will I still need NAT gateway?

Thanks

Is there another workaround? I don't think it makes sense to have a NAT Gateway if the lambda and MSK are in the same subnet. I'm having the exact same issue and due to client's policies we are not allowed to use NAT gateway anyways

Same issue here, lambda does not process any messages, event source mapping is right. lambda and MSK cluster all have matching VPC, subnets and same default VPC that allows all the traffic between them. Tested producer and consumer from an EC2 that has same SG, everything works fine from EC2. However lambda trigger shows me "Last processing result: PROBLEM: Connection error. Please check your event source connection configuration." Lambda integration is one of the main reason we chose MSK over confluent cloud server less option.

Apparently, a public subnet and NAT Gateway are requirements as per the the AWSFeed https://awsfeed.com/whats-new/compute/using-amazon-msk-as-an-event-source-for-aws-lambda. This article has a cloudformation template to create this setup.

Edited by: srirampc on Oct 16, 2020 3:13 PM

I got the same error. MSK trigger for Lambda needs to invoke AWS API for Lambda, so I have added AWS VPC Endpoint (PrivateLink) for Lambda service. Then, the error changed to:
PROBLEM: Connection error. Your VPC must be able to connect to Lambda and STS, as well as Secrets Manager if authentication is required. You can provide access by configuring PrivateLink or a NAT Gateway.
Then, I added also AWS VPC Endpoint (PrivateLink) for STS service and MSK trigger for Lambda has started to work.
I did not add an endpoint for Secrets Manager, as I do not use authentication.

I proposed a documentation improvement here:
https://github.com/awsdocs/aws-lambda-developer-guide/pull/283

I'm hitting the same issue...
"PROBLEM: Connection error. Your VPC must be able to connect to Lambda and STS, as well as Secrets Manager if authentication is required. You can provide access by configuring PrivateLink or a NAT Gateway."

I have an MSK Cluster running in a VPC with only public subnets. Lambda is in the same VPC.

I tried first with security groups as described in the doc - self-referenced in MSK + inbound rule for Lambda SG.
Lambda security group has an inbound rule from anywhere on port 9092.
Then I changed to allow all from everywhere, with no difference.

Is it really a requirement that I need to run MSK in a private subnet with NAT gateway?

I am getting the following error even after following the suggestions
Last processing result: PROBLEM: Connection error. Please check your event source connection configuration.

1. NAT gateway setting
2. self reference rule in MSK security group
3. SG contains proper rules to allow Lambda access

Can anyone help me to fix this issue.

Please note TLS is enabled