By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

"errorMessage": "NoBrokersAvailable" for python kafka producer in a lambda

0

I created the lambda in the same vpc as the amazon msk cluster and has proper execution role. What could be the possible reason for this error ? Also, for the lambda that is using msk as teh trigger , also reporting error " Last processing result: PROBLEM: Connection error. Please check your event source connection configuration. If your event source lives in a VPC, try setting up a new Lambda function or EC2 instance with the same VPC, Subnet, and Security Group settings. Connect the new device to the Kafka cluster and consume messages to ensure that the issue is not related to VPC or Endpoint configuration. If the new device is able to consume messages, please contact Lambda customer support for further investigation."

I can connect and run console kafka producer and consumer from EC2 instance for the same msk cluster.

Also, is there a good example code for using kafka-python in aws lambda for producer ?

Topics
Networking & Content DeliveryServerlessComputeAnalytics
Tags
Amazon VPCAWS LambdaAmazon Managed Streaming for Apache Kafka (Amazon MSK)
Language
English
Mn
asked 9 months ago1124 views
1 Answer
0

Are you configured to connect to the msk cluster from Lambda's?
Are security groups, for example, allowed to connect from Lambda?
https://repost.aws/knowledge-center/lambda-trigger-msk-kafka-cluster

Also, is Lambda connected to a private subnet?
Also, is it possible to connect to an external network from a Lambda in a VPC with a NAT Gateway or similar set up?

profile picture
EXPERT
Riku_Kobayashi
answered 9 months ago
profile pictureAWS
EXPERT
Jonathan_D
reviewed 9 months ago
  • Mn
    9 months ago

    Thank you.

    I added the AWSLambdaMSKExecutionRole in the permission for both producer and consumer lambda. I already created these lambdas in the same vpc , used same security groups .

    For the consumer , now the msk trigger is at least not throwing "connection" error , the status is showing "ok" . But I do not see it is consuming events and sometimes teh trigger drops the connection and shows error , when I reload it few times the connection is successful again.

    For the producer , it still throws 'no broker' error. Any knowledge base to create a producer properly

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content