Browse through the questions and answers listed below or filter and sort to narrow down your results.
Does MSK cluster configuration update will affect existing topic configuration?
For example, if I change partion num in my cluster configuration and apply to the cluster, will it also help me update the existing topic partition number? If not, is there a list of configuration that will not be overriden by cluster configuration change after the resource has been initialized?
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
1
answers
0
votes
4
views
asked 2 months ago
1
answers
0
votes
6
views
asked 3 months ago
Why is there a recommended version for MSK
If you look at the supported version of MSK (https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html) you'll notice that there is a supported version (currently 2.6.2).
Anyone an idea why? What's the drawback with other/newer versions? Less tested? Less stable? ...?
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
1
answers
1
votes
9
views
asked 4 months ago
Does Updating an Active MSK cause data loss
Hi there,
I would like to update an active MSK configuration from 1 to 2. Would that cause any downtime on the current partition as the current data is crucial?
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
3
answers
0
votes
8
views
asked 4 months ago
AWS MSK Connect - Snowflake Plugin (Secrets)
Hi All, I have an MSK Cluster set with 1 broker per 3 AZ's on m5.large and looking to switch our ECS Containers I have running Kafka Connect to using MSK Kafka Connect.
I have a working ECS configuration that runs Kafka Connect with a Snowflake Connector. This user's private key, username, and password per the Snowflake Documentation. When I take the same working configuration and run it with MSK - I get the following error message:
```2022-01-13T07:52:08.000-06:00 [Worker-0576cf5f4676783e1] [SF_KAFKA_CONNECTOR] Exception: Invalid encrypted private key or passphrase
2022-01-13T07:52:08.000-06:00 [Worker-0576cf5f4676783e1] [SF_KAFKA_CONNECTOR] Error Code: 0018
2022-01-13T07:52:08.000-06:00 [Worker-0576cf5f4676783e1] [SF_KAFKA_CONNECTOR] Detail: failed to decrypt private key. Please verify input private key and passphrase.
```
Since my password contains a special character e.g. `!` and the private key contains `/` etc. I am assuming this might need to be escaped?
Appreciate helps.
Thanks
Wayne
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
1
answers
0
votes
14
views
asked 4 months ago
How to add availability zone to existing kafka cluster without downtime?
Hi,
We have an existing AWS managed Kafka cluster which is setup on 2 AZs. How can we add additional AZ to the this cluster without any downtime or recreating the cluster? I couldn't find any documenation about it.
Much appreciated,
jon
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
1
answers
0
votes
3
views
asked 4 months ago
1
answers
0
votes
28
views
asked 5 months ago
1
answers
0
votes
21
views
asked 5 months ago
1
answers
0
votes
0
views
asked 7 months ago
Brokers Connection string doesn't show creating cIuster with IAM Access
I have created an MSK cluster with IAM Access Control as a mechanism for authentication and authorization. I can see the broker nodes for my cluster on MSK Console but when I try to write a message to a topic from my kafka client, it gives me error "Broker may not be available".
When I used the cli command i.e. get-bootstrap-brokers, it shows no value for the "BootstrapBrokerStringSaslIam". It gives me nothing at all. Not even a message telling the status of brokers.
I have followed the official documentation for it https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html
However, I can get the ZooKeeper string for my IAM Access Control cluster and can create kafka topics from my kafka client.
I don't know what's happened to my brokers.
Can anyone tell me what should I do?
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
3
answers
0
votes
12
views
asked a year ago
Amazon MSK questions
A customer has the following questions related to MSK service.
1. What is the good way to manage topics and their settings? (command line tool / UI)
2. Can we adjust settings for existing topic? Can we change replication factor or number of partitions?
3. How to determine the numbers of partitions for the topic? Does it depend on how many brokers do we have?
4. What will happen if we increase the number of brokers? Is it even possible without recreating cluster?
5. We fill-up the Timestamp field in kafka message with timestamps from our devices. Is it ok that these timestamps will not be ordered in time ( we can get timestamps from the past)?
6. Is using protobuf for encode/decode messages a good choice?
7. What do you think about segmentio/kafka-go golang client? Can you advise us on choosing the "best" client?
8. Is it possible to manage commits ourself while using consumer groups?
9. Is there a way to add more disks to a broker instance?
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
1
answers
0
votes
15
views
asked a year ago
Automatically update brokerstring in MSK
A customer is looking to migrate their kafka workload to MSK. they have a question on how to mange the broker strings when a scaling event occurs in the cluster. for example:
the producer has a config which has a property:
`bootstrap.servers`={comma separated string of 3 broker urls}
now a scaling event occurs and now you have another broker.
Now the ugly way is to update the brokerstring url section manually.
the other option is to use a round robin DNS or an ALB and all the clients(producers/consumers) talk to the endpoint and behind the scenes it maintains the broker strings.
though i doubt if second option is doable since MSK is managed. or may be i am understanding it incorrectly?
Looking for some guidance on how this scenario is solved for other customers.
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
1
answers
0
votes
9
views
Connect Amazon MSK cluster to on-premises web server?
I am trying out AWS MSK and created a cluster following the tutorial. What I want to know is, how can I connect my on-premises web sever to the created cluster as a producer. The tutorial only mention how to use AWS MSK with ec2 instance. Is there any other tutorial or blog post that I can refer to?
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
3
answers
0
votes
75
views
asked 3 years ago
1
answers
0
votes
1
views
asked 3 years ago
Updating a configuration
Hello!
I've managed to create a cluster configuration through the CLI (https://docs.aws.amazon.com/cli/latest/reference/kafka/create-configuration.html), and created a cluster that uses that configuration.
How would I go about updating a configuration ("create a new revision?"), and what impact would that have on a cluster?
I think I won't have to update the configuration often, and am mostly looking for the ability to react to operational changes (tune settings for performance, allow/disallow client behaviors such as deleting topics).
I'm not "bound" to using the CLI, so if there is a REST API endpoint I'd be fine with that.
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
2
answers
0
votes
1
views
asked 3 years ago
Clarifying MSK data transfer pricing within region or AZ
Howdy - can someone clarify the MSK data transfer charges within a region or AZ?
The MSK pricing page says "You will pay standard AWS data transfer charges for data transferred in and out of Amazon MSK clusters" with a link to the EC2 on-demand pricing page.
The "Data Transfer within the same AWS Region" section of that page gives the pricing for different combinations of EC2 and various AWS services. Some combinations have free data transfer within a region. Some combos have free data transfer within an AZ. But MSK is not listed there so I can't tell what the pricing is.
Can someone clarify how the pricing between MSK and EC2 works within a region?
I'm particularly interesting in learning about steps I can take to achieve free data transfer between EC2 and the MSK cluster :-)
Thanks! JP
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
3
answers
0
votes
3
views
asked 3 years ago
Backup Kafka topics to S3
MSK is a great service, which considerably reduces operational overhead. One remaining piece is backing up Kafka topics. In particular, when connecting microservices via Kafka this is quite important, i.e. there is high screw up potential if it fails.
Hence, do you have plans to provide backups for MSK, e.g. stream Kafka topics to S3 via Firehose?
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
1
answers
0
votes
2
views
asked 3 years ago
Configure brokers for SimpleAclAuthorizer?
In MSK, is it possible to customize broker properties like `authorizer.class.name` to enable the built-in kafka.security.auth.SimpleAclAuthorizer? And, add a listener with the SASL_PLAINTEXT protocol? If so, where do I set these?
Edited by: kinghuang970 on May 6, 2019 2:34 PM
Accepted AnswerAmazon Managed Streaming for Apache Kafka (Amazon MSK)
1
answers
0
votes
0
views
asked 3 years ago