1 Answers
1
I just started a test cluster on the Confluent cloud and was able to publish messages using the AWS IoT Kafka Action, here the steps I used to set it up:
- Created a VPC with two subnets
- Enabled DNS resolution on my VPC
- Attached VPC Internet Gateway and a created default routing rule on both subnets to the IGW
- Created an AWS IoT VPC Destination using both subnets created before and referenced a Security Group allowing all outbound traffic
- Attached an EIP two both ENI created by the AWS IoT Destination
- Configured the Kafka Action
aws iot get-topic-rule --rule-name KafkaConfluentTest
{
"ruleArn": "arn:aws:iot:eu-west-1:12345678901:rule/KafkaConfluentTest",
"rule": {
"ruleName": "KafkaConfluentTest",
"sql": "SELECT * FROM 'kafka' ",
"description": "",
"createdAt": "2022-08-04T10:24:09+02:00",
"actions": [
{
"kafka": {
"destinationArn": "arn:aws:iot:eu-west-1:12345678901:ruledestination/vpc/8758ddb2-a2d1-4d6d-bfa2-10658b9511d1",
"topic": "test1",
"key": "",
"partition": "",
"clientProperties": {
"acks": "1",
"bootstrap.servers": "pkc-xxxxxx.eu-west-1.aws.confluent.cloud:9092",
"compression.type": "none",
"key.serializer": "org.apache.kafka.common.serialization.StringSerializer",
"sasl.mechanism": "PLAIN",
"sasl.plain.password": "${get_secret('kafka-sasl-passwor
d', 'SecretString', 'arn:aws:iam::12345678901:role/AWSIoTServiceRole')}",
"sasl.plain.username": "${get_secret('kafka-sasl-username', 'SecretString', 'arn:aws:iam::12345678901:role/AWSIoTServiceRole')}",
"security.protocol": "SASL_SSL",
"value.serializer": "org.apache.kafka.common.serialization.ByteBufferSerializer"
}
}
}
],
"ruleDisabled": false,
"awsIotSqlVersion": "2016-03-23"
}
}
After publishing messages using the AWS IoT Test client to my rule, I can see a new entry in the Conflunent.cloud dashboard under Data Integration -> Clients with Client id : aws-iot-rules-engine-a1921787-6654-42aa-9a67-00f9...
If you have things you want me to test in my environment, let me know.
answered 12 days ago
Relevant questions
How to add availability zone to existing kafka cluster without downtime?
Accepted Answerasked 7 months agoAWS MSK kafka-authorizer.log
asked 19 days agoBackup Kafka topics to S3
Accepted Answerasked 3 years agoEndpoint for Kafka Connect
asked a year agoSupport Kafka 3.0.1 to fix KAFKA-13636?
Accepted Answerasked 3 months agoKafka for single packet IoT cluster
asked 3 months agoKafka connect Regex Transform not working in the MSK Connect
asked 6 months agoConnect IOT Core, Kafka Rule to managed Confluent Kafka cluster
asked 12 days agoHow big is the risk when updating the Kafka version?
asked 2 years agoHow to Integrate a Kinesis Data Analytics Flink Application with a Self-Managed Kafka Cluster Running on Amazon EC2 (not Amazon MSK)?
Accepted Answerasked 2 years ago