Skip to content

Replicators in AWS MSK Serverless

0

When moving from AWS MSK Kafka to AWS MSK Kafka serverless, how to manage the existing replicators. Are we need replicators when moving to serverless, will serverless handle replicator/replication by itself.

2 Answers
1

When moving to AWS MSK Kafka Serverless, you no longer need to manually manage replicators as the service handles replication by itself. This allows you to focus more on building applications and less on managing infrastructure.

Please find some documention entries about it:

https://docs.aws.amazon.com/msk/latest/developerguide/serverless.html

https://aws.plainenglish.io/serverless-managed-streaming-for-apache-kafka-msk-a-game-changer-for-real-time-data-processing-22ddc05e5723

EXPERT
answered 2 years ago
EXPERT
reviewed 2 years ago
EXPERT
reviewed 2 years ago
EXPERT
reviewed 2 years ago
0
Accepted Answer

Hi Vinu_129,

Please try this below solution I hope it will help to resolve your issue,

Replication Management in MSK Serverless:

  • AWS MSK Serverless uses Kafka’s native replication for data durability and availability within the same cluster.
  • For details, refer to AWS MSK Serverless Developer Guide.

https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html

Do You Need Replicators in MSK Serverless:

Intra-cluster replication:

  • Kafka's internal replication mechanism continues to function in MSK Serverless.
  • No additional replicators are required for intra-cluster replication.
  • Learn more in the Kafka Documentation on Replication.

https://kafka.apache.org/documentation/#replication

Inter-cluster replication:

  • Tools like Kafka MirrorMaker or Confluent Replicator are still needed for cross-region or inter-cluster replication.
  • MSK Serverless does not handle inter-cluster replication automatically.
  • For more on using MirrorMaker, see Kafka MirrorMaker Documentation.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330

Steps to Transition:

Evaluate your replication needs:

  • Identify essential replication parts and simplify where possible for MSK Serverless.
  • AWS MSK Serverless Considerations.

https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html

Migrate configurations:

  • Set up replication tools to work with MSK Serverless cluster endpoints.
  • Adjust network configurations, access controls, and topic configurations.
  • Configuring Access Control for MSK Serverless.

Test thoroughly:

  • Perform extensive testing to ensure seamless replication setup in MSK Serverless.
  • Testing Amazon MSK Serverless Clusters.
EXPERT
answered 2 years ago
EXPERT
reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.