Lambda MSK trigger not working

0

I have set up an MSK cluster using the defaults, added NAT Gateways to the two subnets MSK is using, and set up all of the permissions on my lambda role described in this article about using MSK as an event source: https://aws.amazon.com/blogs/compute/using-amazon-msk-as-an-event-source-for-aws-lambda/

I know my cluster works with the console tools, so I don't understand what I'm doing wrong. The lambda trigger interface in the UI eventually just says:

Last processing result: PROBLEM: Connection error. Please check your event source connection configuration.

I don't know what to try next to even troubleshoot this, but it's taking a lot of time and is frustrating.

asked 4 years ago6042 views
11 Answers
0

It is very likely the security group attached to the Amazon MSK cluster is blocking the connection. Please create a self-referencing inbound rule on that security group on ports 9092 and 9094. Please check out a lab at https://amazonmsk-labs.workshop.aws/en/msklambda.html that provides step by step instructions.

AWS
answered 4 years ago
0

There was already a self referencing inbound rule for All Traffic on the MSK security group. I also have the lambda security group added to the MSK security group. Not sure what to try next.

answered 4 years ago
0

This error usually indicates a connectivity issue. Can you try spinning up an EC2 instance in one of the subnets as the Amazon MSK cluster, associate the same security group as Amazon MSK and try using one of the Kafka tools to access the Amazon MSK cluster? Please use the appropriate bootstrap broker string depending on whether TLS is enabled for clients on the cluster.
https://docs.aws.amazon.com/msk/latest/developerguide/create-client-machine.html, https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html, https://docs.aws.amazon.com/msk/latest/developerguide/produce-consume.html.

AWS
answered 4 years ago
0

I have done that. I followed the guidance in the article I linked in my first post, which uses an EC2 kafka client to create a new topic and use the console producer/consumer scripts. This all works fine. I have the EC2 instance in it's own security group, but added that group to the MSK security group for all inbound traffic.

answered 4 years ago
0

Hello, just to follow up I was able to resolve this by setting up the proper private subnets->public subnets->NAT Gateways in the vpc the MSK cluster was running in. Thank you for the assistance.

answered 3 years ago
0

Hi,
Just want to know what happens if i have my Lambda in Public subnets and so is the MSK cluster. Will I still need NAT gateway?

Thanks

Kollol
answered 3 years ago
0

Is there another workaround? I don't think it makes sense to have a NAT Gateway if the lambda and MSK are in the same subnet. I'm having the exact same issue and due to client's policies we are not allowed to use NAT gateway anyways

answered 3 years ago
0

Same issue here, lambda does not process any messages, event source mapping is right. lambda and MSK cluster all have matching VPC, subnets and same default VPC that allows all the traffic between them. Tested producer and consumer from an EC2 that has same SG, everything works fine from EC2. However lambda trigger shows me "Last processing result: PROBLEM: Connection error. Please check your event source connection configuration." Lambda integration is one of the main reason we chose MSK over confluent cloud server less option.

Apparently, a public subnet and NAT Gateway are requirements as per the the AWSFeed https://awsfeed.com/whats-new/compute/using-amazon-msk-as-an-event-source-for-aws-lambda. This article has a cloudformation template to create this setup.

Edited by: srirampc on Oct 16, 2020 3:13 PM

answered 3 years ago
0

I got the same error. MSK trigger for Lambda needs to invoke AWS API for Lambda, so I have added AWS VPC Endpoint (PrivateLink) for Lambda service. Then, the error changed to:
PROBLEM: Connection error. Your VPC must be able to connect to Lambda and STS, as well as Secrets Manager if authentication is required. You can provide access by configuring PrivateLink or a NAT Gateway.
Then, I added also AWS VPC Endpoint (PrivateLink) for STS service and MSK trigger for Lambda has started to work.
I did not add an endpoint for Secrets Manager, as I do not use authentication.

I proposed a documentation improvement here:
https://github.com/awsdocs/aws-lambda-developer-guide/pull/283

answered 3 years ago
0

I'm hitting the same issue...
"PROBLEM: Connection error. Your VPC must be able to connect to Lambda and STS, as well as Secrets Manager if authentication is required. You can provide access by configuring PrivateLink or a NAT Gateway."

I have an MSK Cluster running in a VPC with only public subnets. Lambda is in the same VPC.

I tried first with security groups as described in the doc - self-referenced in MSK + inbound rule for Lambda SG.
Lambda security group has an inbound rule from anywhere on port 9092.
Then I changed to allow all from everywhere, with no difference.

Is it really a requirement that I need to run MSK in a private subnet with NAT gateway?

answered 3 years ago
0

I am getting the following error even after following the suggestions
Last processing result: PROBLEM: Connection error. Please check your event source connection configuration.

  1. NAT gateway setting
  2. self reference rule in MSK security group
  3. SG contains proper rules to allow Lambda access

Can anyone help me to fix this issue.

Please note TLS is enabled

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions