By using AWS re:Post, you agree to the Terms of Use
/MSK Custom Configuration using Cloudformation/

MSK Custom Configuration using Cloudformation

0

Hi AWS Users,

I am trying to spin up a MSK cluster with a custom MSK configuration using my serverless app. I wrote the cloudformation template for the generation of the MSK Cluster and was able to successfully bring it up. I recently saw that AWS added cloudformation template of AWS::MSK::Configuration. [1] I was trying that out to create a custom configuration. The Configuration requires a ServerPropertieskey that is usually a PlainText in AWS console.

An example of Server Properties:

auto.create.topics.enable=true
default.replication.factor=2
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=10
num.replica.fetchers=2
replica.lag.time.max.ms=30000
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
unclean.leader.election.enable=true
zookeeper.session.timeout.ms=18000

AWS::MSK::Configuration accepts base64 (api functionality) and I have been trying to implement this. I am using the cloudformation Fn::Base64 functionality. e.g:

Resources:
  ServerlessMSKConfiguration:
    Type: AWS::MSK::Configuration
    Properties:
      ServerProperties:
        Fn::Base64: auto.create.topics.enable=true

This gives me back a 400 error during deploy.

Resource handler returned message: "[ClientRequestToken: xxxxx] Invalid request body (Service: Kafka, Status Code: 400, Request ID: 1139d840-c02d-4fdb-b68c-cee93673d89d, Extended Request ID: null)" (RequestToken: xxxx HandlerErrorCode: InvalidRequest)

Can someone please help me format this ServerProperties properly, not sure how to give the proper base64 string in the template. Any help is much appreciated.

[1] - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html

1 Answers
0

Here's one that works for me. It took some trial and error to figure out that Name and Description only accept some limited special characters, and the KafakVersionsList couldn't be a variable. And this did not require !Base64 to create the configuration.

  MskConfiguration:
    Type: "AWS::MSK::Configuration"
    Properties:
      Description: !Sub "${EnvironmentName}-${EnvironmentShard}-msk-cluster-configuration"
      Name: !Sub "${EnvironmentName}-${EnvironmentShard}-msk-cluster-configuration"
      KafkaVersionsList: [2.8.1]
      ServerProperties: |
        auto.create.topics.enable=false
        default.replication.factor=2
answered 17 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions