Does Updating an Active MSK cause data loss

0

Hi there, I would like to update an active MSK configuration from 1 to 2. Would that cause any downtime on the current partition as the current data is crucial?

profile picture
질문됨 2년 전1465회 조회
4개 답변
1
수락된 답변

Hi there, are you looking to update MSK cluster configuration and apply a new revision to the cluster? Any updates or version upgrade wouldn't cause any data loss. When an update is triggered, cluster will undergo a rolling reboot i.e change/update will be applied on one broker at a time and rebooted.

During this time one broker will not be available and all the partition leadership this broker contains will be reassigned to other active brokers in the cluster at that time and those active brokers continue to accept client IO. As a best practice to avoid any disruption in client IO make sure to have RF of topics atleast equal to number of AZs and minISR = RF-1.

AWS
지원 엔지니어
답변함 2년 전
AWS
전문가
검토됨 2년 전
0

Thanks for the answer. The update was completed but partition counts didn't increase when I ran this command ./kafka-topics.sh --describe --bootstrap-server b-2.non-prod-***:9092 --topic. The config file should the increment though. Any idea on what am missing?

profile picture
답변함 2년 전
0

Hi again, based on question I'm assuming the config change that was made is related to number of partitions. Correct me if I'm wrong. If that is indeed the change you made, it applies only to the newly 'auto created' topics after the change has been made. This config doesn't change partition count of the existing topics. If you would like to change the partition count of existing topic, you can do so by running below command. Hope that answers!

./kafka-topics.sh --bootstrap-server <brokers> --alter --topic <topic> --partitions <new count>

AWS
지원 엔지니어
답변함 2년 전
0

follow up to build highly available clusters, won't happen data lost during patching upgrade

Use the following recommendations so that your MSK cluster can be highly available during an update (such as when you're updating the broker type or Apache Kafka version, for example) or when Amazon MSK is replacing a broker.

Set up a three-AZ cluster.

Ensure that the replication factor (RF) is at least 3. Note that a RF of 1 can lead to offline partitions during a rolling update; and a RF of 2 may lead to data loss.

Set minimum in-sync replicas (minISR) to at most RF - 1. A minISR that is equal to the RF can prevent producing to the cluster during a rolling update. A minISR of 2 allows three-way replicated topics to be available when one replica is offline.

Ensure client connection strings include at least one broker from each availability zone. Having multiple brokers in a client's connection string allows for failover when a specific broker is offline for an update. For information about how to get a connection string with multiple brokers, see Getting the bootstrap brokers for an Amazon MSK cluster.

AWS
답변함 7달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠