MSK upgrade from 2.6.x to 3.5.x

0

Our MSK cluster is running on version 2.6 and we are attempting to upgrade to 3.5. According to the Kafka documentation, we need to set the inter.broker.protocol.version property. https://kafka.apache.org/35/documentation.html#upgrade_3_5_0.

Question is how do we set the value of this property (or is it done automatically by AWS)?

asked 13 days ago91 views
1 Answer
1

💡 For upgrading your Apache Kafka version on Amazon MSK, simply refer to the guide Updating the Apache Kafka Version. This resource provides detailed instructions on how to perform the update using either the AWS Management Console or the AWS CLI.

If you prefer using the CLI, here's how you can proceed:

  1. Execute the following command to retrieve a list of Kafka versions compatible with your current setup:
   aws kafka get-compatible-kafka-versions --cluster-arn ClusterArn

ℹ️ This command will produce an output similar to the example below, detailing both your current version and the versions to which you can upgrade:

   {
      "CompatibleKafkaVersions": [
          {
              "SourceVersion": "2.2.1",
              "TargetVersions": [
                  "3.3.1",
                  "3.4.1",
                  "3.4.1.1",
                  "3.5.1"
              ]
          }
      ]
   }

ℹ️ This information will help you determine the target version for upgrading your MSK cluster.

  1. Select your desired target version from the list obtained in the previous step. Then, update your cluster to this version by running the following command:
   aws kafka update-cluster-kafka-version --cluster-arn ClusterArn --current-version Current-Cluster-Version --target-kafka-version TargetVersion

ℹ️ Alternatively, you can also perform this update through the AWS Management Console if you prefer a graphical interface.


🔑 Additionally, it is highly recommended to adhere to the Best Practices for Version Upgrades outlined by AWS. Following these guidelines will help ensure a smooth and successful upgrade of your MSK cluster.

profile picture
EXPERT
answered 13 days ago
  • Thanks for this answer! I'm comfortable with the MSK upgrade process in general, as I've done this before. I was specifically wanting some details around the property that was advised to be set in the official kafka documentation. The AWS docs don't mention this property.

    I didn't ask this in the original question, but I suppose more broadly, does AWS apply this setting as part of the upgrade process? If I try to apply it, it's flagged as an error stating it's an invalid property.

  • If I am not wrong, AWS MSK handles the inter.broker.protocol.version setting automatically as part of the upgrade process, so you do not need to set it manually. If you attempt to apply it, MSK will reject it as an invalid property because it manages such configurations internally. This is to simplify the operation and ensure cluster stability during upgrades. This is why you see an error when trying to manually set this property; AWS automates these configurations as part of the managed service's upgrade process.

  • Thanks so much! This is exactly the answer I was hoping to get :D

  • Great to hear that. Would you mind validating the answer to help others in the future?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions