Can MSK Connect be made rack-aware?

0

We have a Kafka cluster in Amazon MSK that has 3 brokers in different availability zones of the same region. I want to set up a Kafka Connect connector that backs up all data from our Kafka brokers to Amazon S3, and I'm trying to do it with MSK Connect.

I set up Confluent's S3 Sink Connector on MSK Connect and it works - everything is uploaded to S3 as expected. The problem is that it costs a fortune in data transfer charges - our AWS bills for MSK nearly double whenever the connector is running, with EU-DataTransfer-Regional-Bytes accounting for the entire increase.

It seems that the connector is pulling messages from all three of our brokers, i.e. from three different AZs, and so we're getting billed for inter-AZ data transfer. This makes sense because by default it will read a partition from that partition's leader, which could be any of the three brokers. But if we were creating a normal consumer, not a connector, it would be possible to restrict the consumer to read all partitions from a specific broker:

"client.rack" : "euw1-az3"

☝️ For a consumer in the euw1-az3 AZ, this setting makes the consumer read all partitions from the local broker, regardless of the partitions' leader - which avoids the need for inter-AZ data transfer and brings the bills down massively.

My question is, is it possible to do something similar for a Kafka Connector? What config setting do I have to pass to the connector, or the worker, to make it only read from one specific broker/AZ? Is this possible with MSK Connect?

已提问 2 年前740 查看次数
2 回答
0

Hi there, in order to fetch from a nearest replica consumer side config client.rack must be set as you might be aware. When using a Sink connector, it acts as a consumer to Kafka topic so Kafka connect allows you to specify all consumer config by using prefix 'consumer.*' However in MSK any configuration property that starts with the consumer. prefix is allowed only if it starts with the prefixes mentioned here https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-workers.html#msk-connect-custm-worker-config. Hope this helps!

AWS
支持工程师
已回答 2 年前
0
profile picture
已回答 2 个月前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则