Can MSK Connect be made rack-aware?
We have a Kafka cluster in Amazon MSK that has 3 brokers in different availability zones of the same region. I want to set up a Kafka Connect connector that backs up all data from our Kafka brokers to Amazon S3, and I'm trying to do it with MSK Connect.
I set up Confluent's S3 Sink Connector on MSK Connect and it works - everything is uploaded to S3 as expected. The problem is that it costs a fortune in data transfer charges - our AWS bills for MSK nearly double whenever the connector is running, with EU-DataTransfer-Regional-Bytes
accounting for the entire increase.
It seems that the connector is pulling messages from all three of our brokers, i.e. from three different AZs, and so we're getting billed for inter-AZ data transfer. This makes sense because by default it will read a partition from that partition's leader, which could be any of the three brokers. But if we were creating a normal consumer, not a connector, it would be possible to restrict the consumer to read all partitions from a specific broker:
"client.rack" : "euw1-az3"
☝️ For a consumer in the euw1-az3 AZ, this setting makes the consumer read all partitions from the local broker, regardless of the partitions' leader - which avoids the need for inter-AZ data transfer and brings the bills down massively.
My question is, is it possible to do something similar for a Kafka Connector? What config setting do I have to pass to the connector, or the worker, to make it only read from one specific broker/AZ? Is this possible with MSK Connect?
Hi there, in order to fetch from a nearest replica consumer side config client.rack must be set as you might be aware. When using a Sink connector, it acts as a consumer to Kafka topic so Kafka connect allows you to specify all consumer config by using prefix 'consumer.*' However in MSK any configuration property that starts with the consumer. prefix is allowed only if it starts with the prefixes mentioned here https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-workers.html#msk-connect-custm-worker-config. Hope this helps!
Relevant questions
MSK Connect - Exposing Connector Specific Metrics
asked 5 months agoMSK Connect Debezium Connector using AWS Glue Schema Registry for AVRO serialization
asked 5 months agoKafka connect Regex Transform not working in the MSK Connect
asked 5 months agoMSK connect connector deployment
asked 12 days agoMSK Connect - Failed to create using in-built AWSServiceRoleForKafkaConnect role
asked 5 months agoCan MSK Connect be made rack-aware?
asked 5 months agoEndpoint for Kafka Connect
asked 9 months agoSet consumer.isolation.level in MSK Connect (sink connector)
asked 12 days agoHow to Integrate a Kinesis Data Analytics Flink Application with a Self-Managed Kafka Cluster Running on Amazon EC2 (not Amazon MSK)?
Accepted Answerasked 2 years agoHow to reset offset of a consumer group created through MSK connect
asked 2 months ago