How much do Kafka transactions cost you on Amazon MSK Express brokers? A measured deep dive

Kafka transactions let you send multiple messages as one atomic batch. Each batch adds a commit step, but quantified data on what that commit costs in throughput is hard to find. I measured it directly on an Amazon MSK Express broker cluster.

The goal was to answer two questions:

1. With transactions, does a broker hit some other limit before reaching its network throughput quota? (If so, sizing by the quota alone under-provisions the cluster.)
2. If it does hit a limit, does scaling the broker size fix it?

Test environment

• MSK Express cluster with 3x express.m7g.large, then the same measurements repeated after updating brokers to 3x express.m7g.xlarge
• Official throughput quota for the initial configuration: 46.8 MBps sustained aggregate, 70.2 MBps maximum (throttle point)
• Two producer hosts and one consumer host in the same VPC
• 4KB messages by default; transaction size (messages per batch) varied across 5 / 20 / 80 / 320
• Non-transactional sends measured under identical conditions as the baseline
• Custom measurement tool — kafka-producer-perf-test.sh only supports time-based commits and cannot pin an exact "commit every N messages" batch size

Measurement validity checks: zero send errors across all runs, no tool bottleneck (non-transactional mode reached 98% of the quota), client-side numbers matched CloudWatch broker metrics within ±4%, and producer host CPU peaked at 18%.

Result 1: throughput drops sharply as transaction batches get smaller

3x express.m7g.large, 4KB messages:

Send mode	Max throughput	vs. non-transactional
Non-transactional	68.5 MBps	100%
Transactions, 320-message batches	43.9 MBps	64%
Transactions, 80-message batches	33.6 MBps	49%
Transactions, 20-message batches	26.9 MBps	39%
Transactions, 5-message batches	8.3 MBps	12%

Non-transactional sends reached 98% of the maximum quota (70.2 MBps). In other words, without transactions you can trust the official throughput quotas for sizing.

But with 5-message batches, the same cluster stalls at 12% of the limit. The network was idle and producer host CPU was under 20%, yet throughput would not increase. The "resources are free but throughput won't grow" situation reproduced exactly.

Result 2: the bottleneck is the commit — a fixed cost of about 55 ms each

Breaking down where the producer spends time using its internal metrics (txn-commit-time-ns-total and friends):

• Sending a message batch and getting the response: 5-8 ms on average
• One commit: about 55 ms (roughly 10x the send)
• With a single producer, effectively 100% of wall-clock time is spent waiting on commits

Nothing reduced this 55 ms:

Attempt	Time per commit
Spread across 24 partitions	55.5 ms
Concentrated on 1 partition	55.3 ms
Brokers replaced with 2x size	57.1 ms

On commit, the broker writes the transaction state to the internal topic (__transaction_state), replicates it across three brokers, and writes commit markers to each partition. That round trip is the fixed cost, and the more often you commit (smaller batches), the more this wait drags down throughput. This explains Result 1.

Result 3: producer parallelism works — up to a point

A single producer can do nothing while waiting on a commit, so multiple producers can fill each other's wait time. Measured with 20-message batches, 4KB messages:

Producers (total)	Throughput	Wait per commit
4	7.9 MBps	35 ms
16	27.4 MBps	42 ms
64	37.3 MBps (peak)	~114 ms
256	29.3 MBps (-21% from peak)	up to 1.5 s

Scaling is near-linear up to 16, peaks around 64, and beyond that commit waits grow while throughput actually decreases. The broker's commit-processing capacity is the counter; adding producers past it only makes the line longer.

To confirm the limit is on the broker side, I modified the producer to pre-send the next batch from another producer while one waits on its commit (latency hiding). Total throughput did not increase at all. If no client-side cleverness helps, the ceiling is on the broker.

Result 4: doubling broker size roughly doubles transactional throughput

After replacing brokers with xlarge and repeating all measurements:

Metric	3x large	3x xlarge	Ratio
Non-transactional max	68.5 MBps	136.4 MBps	2.0x
Transactions, 5-message batches	8.3 MBps	19.3 MBps	2.3x
Transactions, 20-message batches	26.9 MBps	63.9 MBps	2.4x
Transactions, 320-message batches	43.9 MBps	90.1 MBps	2.1x
Peak commits per second	~477	~820-990	~2x

The transaction bottleneck scales away with broker size. This is the most important sizing conclusion of the test.

One exception: the 55 ms per commit did not change with bigger brokers. Scaling up increases how many commits can be processed concurrently, not how fast one commit completes. A single producer therefore gets no faster on bigger brokers — total throughput must come from producer parallelism.

Result 5: on small brokers, CPU fills up before the network does

Broker CPU (CloudWatch, CpuUser + CpuSystem, average of 3 brokers):

Workload	Ingress	Broker CPU
Non-transactional max	66 MBps	47%
Transactions, 320-message batches	42 MBps	40%
Transactions, 20-message batches	36-39 MBps	69-73%
Transactions, 5-message batches	12 MBps	73%
1KB messages + 5-message batches	3 MBps	79%

The last row is the key finding. 3 MBps of ingress, and CPU is at 79%. Commits consume CPU per event, not per byte, so a workload of small messages committed in small batches exhausts the broker at a point that network metrics cannot predict.

The AWS-recommended CPU ceiling is 60%, and much of the transactional range on large brokers exceeded it. On xlarge, CPU always had headroom (under 45%). If you run transactions on small brokers, a 60% CPU alarm is mandatory — and remember that Express can apply protective throttling above 60% CPU.

Result 6: the consumer side is effectively free

Measuring the cost of isolation.level=read_committed on consumers of transactional messages: the difference from a regular read was 0.3%, within noise. The cost of transactions concentrates on the producer (commit) side; the consumer side is not a concern.

Recommendation: a sizing formula for transactional workloads

One network calculation is not enough. Compute both of the following and follow whichever demands more brokers:

• Calculation 1 (existing): required MBps ÷ per-broker throughput quota
• Calculation 2 (new): required commits per second ÷ per-broker commit capacity
  • Required commits per second = messages per second ÷ transaction size
  • Per-broker commit capacity (measured here): large ~160/s, xlarge ~270-330/s

For example, 20,000 messages per second in 10-message batches requires 2,000 commits per second. Three large brokers top out around 480/s — no amount of spare network capacity helps.

When you need more performance, try in this order:

1. Increase the transaction size (zero cost, biggest effect — 5 to 320 gave 5.3x). The trade-offs: longer wait per commit (measured 55 to 234 ms) and a larger re-send scope on failure. The right answer is the largest batch your business requirements allow.
2. Parallelize producers. Not indefinitely — the signal to stop is when throughput stops growing while commit waits keep increasing (around 60 total in this environment).
3. If still short, scale up the broker size. The effect is roughly proportional to size.

Note (latency requirements): The ~55 ms per commit could not be reduced by any means, and transaction-related broker configurations on Express are read-only, leaving no tuning room. If you have a requirement like "commit completes within 55 ms of send," reconsider whether transactions are the right tool.

Operational tip (transactional.id): When running transactional producers on multiple hosts, overlapping transactional.id values cause mutual fencing (ProducerFencedException). My first measurement run had to be discarded entirely because of this mistake. Guarantee a unique ID per producer instance.

Related information

• Express broker throughput quotas
• Best practices for Express brokers (CPU monitoring)
• Express broker configurations (read-only properties)
• Apache Kafka documentation — transactions
• KIP-98 (Exactly Once Delivery and Transactional Messaging)