- Newest
- Most votes
- Most comments
I apologize, but I need to respectfully disagree with your assessment based on the information provided by AWS. According to AWS, Express brokers for Amazon MSK are designed to offer significant performance advantages over Standard brokers, including up to 3 times more throughput per broker. However, it's important to note that performance can vary depending on specific use cases and configurations.
The 3x throughput claim is specifically mentioned for certain instance types. For example, on m7g.16xlarge instances, Express brokers can safely handle up to 500 MBps ingress compared to 153.8 MBps on equivalent Standard brokers. This increased throughput is achieved through opinionated settings and resource isolation.
That being said, your experience of slower performance, especially on the publishing side, is concerning and unexpected based on AWS's claims. The performance advantage of Express brokers is supposed to be particularly noticeable in scenarios with high throughput requirements.
It's worth considering a few factors that might affect your specific case:
-
Instance size: Express brokers come in different sizes (large, 4xlarge, and 16xlarge), and the performance benefits may be more pronounced on larger instances.
-
Workload characteristics: Your scenario of "very few, but extremely busy publishers" might not be the optimal use case for Express brokers. The performance advantages might be more noticeable with a different workload pattern.
-
Configuration: Express brokers come preconfigured with certain best practices, which might not be optimal for your specific use case.
-
Kafka version: Express brokers are supported on Apache Kafka version 3.6, so ensure you're using a compatible version.
Given the discrepancy between your experience and AWS's claims, it would be advisable to contact AWS support for a detailed analysis of your specific setup and workload. They should be able to provide more insights into why you're not seeing the expected performance improvements and potentially help optimize your configuration for better results.
Sources
Amazon MSK Express brokers - Amazon Managed Streaming for Apache Kafka
Express brokers for Amazon MSK: Turbo-charged Kafka scaling with up to 20 times faster performance | AWS Big Data Blog
Introducing Express brokers for Amazon MSK to deliver high throughput and faster scaling for your Kafka clusters | AWS News Blog
Express brokers for Amazon MSK is now generally available - AWS
Relevant content
- asked 2 years ago
- asked 9 months ago
- asked 3 years ago
AWS OFFICIALUpdated 3 years ago
I'd recommend to focus on your specific use case and maybe help you optimize your workload, but for that, could you please share your setup? specifically, what are the client side config on the producer side, how many producers and what is the cluster setup (number of brokers, instance type, worker config, any additionally applied dynamic configs)? Would also make sense to include your non-Express setup to understand what could cause 5 times slower ingestion rate. There is always will be exceptions to the rule, that's why you won't find any proper benchmarking on Kafka.