Can MSK Connect be made rack-aware?

Question

We have a Kafka cluster in Amazon MSK that has 3 brokers in different availability zones of the same region. I want to set up a Kafka Connect connector that backs up all data from our Kafka brokers to Amazon S3, and I'm trying to do it with MSK Connect.

I set up Confluent's S3 Sink Connector on MSK Connect and it works - everything is uploaded to S3 as expected. The problem is that it costs a fortune in data transfer charges - our AWS bills for MSK nearly double whenever the connector is running, with `EU-DataTransfer-Regional-Bytes` accounting for the entire increase.

It seems that the connector is pulling messages from all three of our brokers, i.e. from three different AZs, and so we're getting billed for inter-AZ data transfer. This makes sense because by default it will read a partition from that partition's leader, which could be any of the three brokers. But if we were creating a normal consumer, not a connector, it would be possible to restrict the consumer to read all partitions from a specific broker:

```
"client.rack" : "euw1-az3"
```

☝️ For a consumer in the euw1-az3 AZ, this setting makes the consumer read all partitions from the local broker, regardless of the partitions' leader - which avoids the need for inter-AZ data transfer and brings the bills down massively.

My question is, is it possible to do something similar for a Kafka Connector? What config setting do I have to pass to the connector, or the worker, to make it only read from one specific broker/AZ? Is this possible with MSK Connect?

Harshith_M · Answer

Hi there, in order to fetch from a nearest replica consumer side config client.rack must be set as you might be aware. When using a Sink connector, it acts as a consumer to Kafka topic so Kafka connect allows you to specify all consumer config by using prefix 'consumer.*' However in MSK any configuration property that starts with the consumer. prefix is allowed only if it starts with the prefixes mentioned here https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-workers.html#msk-connect-custm-worker-config. Hope this helps!

JohnPreston · Answer

See https://aws.amazon.com/blogs/big-data/reduce-network-traffic-costs-of-your-amazon-msk-consumers-with-rack-awareness/ as well to make sure that the cluster allows for it.

Can MSK Connect be made rack-aware?

Relevant content