self managed Kafka and Kafka trigger Lambda Connection error

Question

![Enter image description here](/media/postImages/original/IMCeY5Ag9jQmaOuMXD84KCWg)

I want to ask how to connect lambda trigger kafka with self managed kafka broker
i do running kafka server and open access to the advertised address 'ec2-18-141-185-236.ap-southeast-1.compute.amazonaws.com:9092'

but when i wan to listen it on the lambda it not receive any message but the trigger status become Connection error

this is  my setup connection

my trigger kafka hosted on public vpc subnet so it can access internet and should be access my kafka public address : 'ec2-18-141-185-236.ap-southeast-1.compute.amazonaws.com:9092'

rePost-User-0695010 · Answer

To connect Lambda to a self-managed Kafka ka broker, you need to configure several things:

**    Kafka Broker Configuration:**
```
# server.properties
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://ec2-18-141-185-236.ap-southeast-1.compute.amazonaws.com:9092
```

**
    Lambda Function VPC Configuration:
**

```
Resources:
  MyLambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      VpcConfig:
        SecurityGroupIds:
          - sg-xyz                    # fill appropriate value
        SubnetIds:
          - subnet-xyz     # fill appropriate value

```

**    Security Group Rules:**

```
# Lambda Security Group
- Outbound: Allow TCP 9092 to Kafka Security Group

# Kafka Security Group
- Inbound: Allow TCP 9092 from Lambda Security Group
```

**    Lambda Function Configuration:
**

```
def lambda_handler(event, context):
    kafka_bootstrap_servers = ['ec2-18-141-185-236.ap-southeast-1.compute.amazonaws.com:9092']
    kafka_topic = 'your-topic'
    
    consumer_config = {
        'bootstrap.servers': ','.join(kafka_bootstrap_servers),
        'group.id': 'lambda-consumer-group',
        'auto.offset.reset': 'earliest'
    }
```

** Lambda Event Source Mapping:**

```
{
  "EventSourceArn": "self-managed",
  "FunctionName": "your-lambda-function",
  "SourceAccessConfigurations": [
    {
      "Type": "VPC_SUBNET",
      "URI": "subnet-*" # fill appropriate value
    },
    {
      "Type": "VPC_SECURITY_GROUP",
      "URI": "sg-" # fill appropriate value
    }
  ],
  "SelfManagedEventSource": {
    "Endpoints": {
      "KAFKA_BOOTSTRAP_SERVERS": [
        "ec2-18-141-185-236.ap-southeast-1.compute.amazonaws.com:9092"
      ]
    }
  }
}

```

**  Check Network Routes:    **
# Ensure VPC has:
- Internet Gateway (for public subnets)
- NAT Gateway (for private subnets)
- Proper route tables

**Troubleshooting Steps:**

Verify Connectivity:
``` 
# From EC2 in same VPC as Lambda
telnet ec2-18-141-185-236.ap-southeast-1.compute.amazonaws.com 9092
```

# Check Kafka logs
```
tail -f /var/log/kafka/server.log
```
** Test Kafka Connection:**

```
from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'your-topic',
    bootstrap_servers=['ec2-18-141-185-236.ap-southeast-1.compute.amazonaws.com:9092'],
    group_id='test-group'
)

```

**    Lambda IAM Role:**

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DeleteNetworkInterface"
            ],
            "Resource": "*"
        }
    ]
}

```

Enable Lambda VPC Flow Logs to debug connectivity issues.

**Remember:**

Ensure broker advertised.listeners matches the public DNS
    Security groups must allow traffic both ways
    Lambda needs proper VPC access
    Consider using TLS for security
    Monitor Lambda timeout settings

rePost-User-0695010 · Answer

There are below common Issues you need to check:

DNS Resolution
    Route Table configuration
    NACL rules
    Security Group stateful rules
    Kafka broker network settings
    Lambda VPC subnet public IP settings
    Internet Gateway attachment

Also remember to:

Check CloudWatch Logs for detailed error messages
    Verify all network paths
    Test with a simple consumer script first
    Increase Lambda timeout for testing
    Enable detailed monitoring temporarily

you can use AWS Network Reach-ability Analyzer to check connectivity between Lambda and your Kafka broke
VPC > Network Analyzer > Network Reachability Analyzer > Create analysis

Note: You'll need the ENI (Elastic Network Interface) IDs for both your Lambda function and Kafka instance to create the analysis.

https://docs.aws.amazon.com/vpc/latest/network-access-analyzer/getting-started.html

self managed Kafka and Kafka trigger Lambda Connection error

Ensure VPC has:

Check Kafka logs

Add your answer

Relevant content