Why raising socket.receive.buffer.bytes on Amazon MSK does not improve producer throughput

Symptoms

You set socket.receive.buffer.bytes higher than the default (102400), but producer throughput does not improve. When you check from the producer host with ss, you see three things together:

• The receive window the broker advertises (snd_wnd on the producer side) stops at a value far below what you configured.
• Two different values produce identical results. For example, applying 1 MiB and then 2 MiB leaves snd_wnd at the same value.
• Most bytes sent are accounted for as rwnd_limited. In other words, the producer waits because of the receiver's window, not because of congestion control or its own send buffer.

Diagnostic steps

1. First confirm the configuration was actually applied.

Use DescribeConfigurationRevision to check the contents of the configuration revision, and verify the cluster is using that revision. This is a static configuration, so it does not take effect until the broker restart completes.

2. Measure the actual window from the producer host.

ss -itn dst <broker-ip>

Check snd_wnd and rwnd_limited on the TCP info line of the output.

ESTAB 0 0 <producer-ip>:<port> <broker-ip>:9092
     cubic wscale:8,8 rtt:1.2/0.1 ... snd_wnd:204044
     bytes_sent:... rwnd_limited:12345678ms(99.1%) sndbuf_limited:0ms(0.0%)

A high rwnd_limited percentage means the bottleneck is the receiver's window.

Note: The numbers above are illustrative. The actual ceiling can differ by broker type and generation, so always measure in your own environment.

3. Compare measurements using two different large values.

If you apply 1 MiB and then 2 MiB and snd_wnd comes out the same, you have hit a ceiling rather than a failed configuration. If the values differ, the ceiling is not your problem and you should look elsewhere.

4. Rule out the producer's send buffer.

The producer's send.buffer.bytes sets SO_SNDBUF on the client socket. The TCP receive window is a value the receiver (the broker) advertises in its ACKs based on the free space in its own receive buffer, so the client's send buffer size does not shrink the window the broker advertises. Raising the producer's send.buffer.bytes will not change snd_wnd, so do not spend time on it.

Root cause

socket.receive.buffer.bytes sets the SO_RCVBUF of the broker's socket server sockets. The official documentation defines only that the minimum value is -1 and that -1 uses the OS default; it does not state an upper bound.[1][2]

On Linux, the maximum value you can set through SO_RCVBUF is bounded by the kernel parameter net.core.rmem_max. socket(7) states that for SO_RCVBUF the "maximum allowed value is set by the /proc/sys/net/core/rmem_max file",[3] and tcp(7) documents the same constraint.[4]

As a result, when you specify a value that exceeds net.core.rmem_max, the kernel silently clamps it to the ceiling. If both 1 MiB and 2 MiB exceed the ceiling, the effective buffer size is the same, and so is the advertised receive window derived from that buffer. This is why two different values produce identical results.

It also explains why the advertised receive window does not exactly match the buffer size. The receive buffer space is shared between the TCP window and an application buffer, and the kernel reserves part of it as the application buffer to absorb scheduling and application latency.[4]

Because Amazon MSK is a managed service, you cannot view or modify net.core.rmem_max and net.ipv4.tcp_rmem on the broker hosts, and those values are not published. This means you cannot calculate your way around the ceiling — you have to measure it.

Resolution

1. Set socket.receive.buffer.bytes to -1.

With -1, MSK uses the OS default and does not pin SO_RCVBUF explicitly.[1][2] Linux receive buffer auto-tuning (tcp_moderate_rcvbuf, enabled by default) then sizes the buffer to match the throughput the path requires, up to the max value of net.ipv4.tcp_rmem — a ceiling separate from the net.core.rmem_max limit that applies to an explicitly set SO_RCVBUF.[4]

As a result, -1 can lead to a larger advertised window than explicitly configuring a large value.

2. Trying a bigger number does not help.

Any value you specify explicitly is subject to the same net.core.rmem_max ceiling, regardless of its size. Once you are already above the ceiling, raising 1 MiB to 8 MiB changes neither the effective buffer nor the advertised window.

3. Applying the change involves a broker restart.

socket.receive.buffer.bytes is a static configuration. On Express brokers it is classified as static, so applying it requires a broker restart.[2] On standard brokers, updating the cluster with a custom configuration also causes Amazon MSK to perform a rolling restart when necessary.[1] Plan for your operational window.

4. Measure again with the same ss command after the change.

Check whether snd_wnd increases and the rwnd_limited percentage drops. Both indicators must improve together for the change to be effective.

5. Understand what auto-tuning assumes.

Auto-tuning grows the buffer in response to the RTT and bandwidth-delay product of actual traffic. If connections are short-lived or carry little data, the buffer will not grow much, so the benefit of -1 is most pronounced on sustained, high-throughput connections.