If you allow Amazon MSK broker private IPs one by one in security groups or firewalls: what happens to broker IPs when you remove or add brokers, and allow rules that don't depend on IPs

Symptom / situation

This question comes up often in environments like the following:

• Your company security policy doesn't allow another security group as the source or destination of a security group rule.
• Security group rules don't accept domain names, so you registered each broker private IP as a /32 entry.
• To optimize cost, you plan to reduce the cluster from 6 brokers to 3, and you want to confirm two things:
  1. Do the remaining brokers keep their IPs after the reduction?
  2. If you later scale back to 6 brokers, do the new brokers come back with the same IPs and broker IDs as before?

Cause -- broker private IPs are not fixed values

The MSK documentation does not guarantee that broker private IPs stay fixed. Instead, Best practices for Apache Kafka clients states that brokers go offline for both planned and unplanned events, such as upgrades, patching, hardware failure, and network issues, and that Kafka clients must handle broker fail-over gracefully. When a broker is replaced with new infrastructure, its private IP can change.

The answers to the two questions are therefore:

Question	Answer	Basis
Do the remaining brokers keep their IPs after the reduction?	Don't assume they keep the same IPs afterward	The documentation doesn't guarantee that broker IPs are retained, and the remaining brokers can still be replaced later during events such as patching or failure recovery
Do added brokers come back with the previous IPs and IDs?	Don't assume they do	Brokers added by expansion are new brokers with new elastic network interfaces (ENIs). The security group change documentation tells you to update "the ENIs of the new brokers" separately after adding brokers. Nothing in the documentation says the IPs or IDs of removed brokers are reused

One more point to note: you don't choose which brokers get removed. According to Remove a broker from an Amazon MSK cluster, MSK picks the brokers that don't have any partitions on them (except for canary topics). If you want to keep brokers 1, 2, and 3, move the user topic partitions to those brokers before the removal. The example in the documentation shows the same approach: it consolidates partitions onto 3 of 6 brokers with kafka-reassign-partitions.sh --broker-list 1,2,3.

Note: Broker removal is supported one broker per Availability Zone, on M5 and M7g based provisioned clusters, with Kafka 2.8.1 and later (including KRaft mode). On a three-AZ cluster, a single removal operation removes three brokers.

Diagnosis -- check the current broker IPs

Use list-nodes to check whether the IPs you registered still match the actual brokers. The output shows the broker ID, IP address, ENI, and subnet for each broker.

aws kafka list-nodes \
  --cluster-arn <cluster ARN> \
  --query 'NodeInfoList[].BrokerNodeInfo.[BrokerId,ClientVpcIpAddress,AttachedENIId,ClientSubnet]' \
  --output table

Compare this output with the IP list in your security groups or firewalls. If you run the command before and after each removal or expansion and keep the results, you can see right away which entries need to change.

Resolution -- allow a unit that doesn't change instead of individual IPs

A security group rule accepts three kinds of source or destination: IP addresses or CIDR blocks, managed prefix lists, and other security groups (Security group rules). If you can't use security group references, your options are CIDR blocks and prefix lists.

1. Allow the broker subnet CIDRs (recommended)

Broker ENIs are created in the client subnets that you specified when you created the cluster. The ClientSubnet value in the list-nodes output above is that subnet. Even when brokers are replaced or added, their IPs are assigned from these subnet ranges. If you allow only the ports you use to the subnet CIDRs, you don't need to edit the rules on every IP change, removal, or expansion.

# Check the cluster's client subnets
aws kafka describe-cluster --cluster-arn <cluster ARN> \
  --query 'ClusterInfo.BrokerNodeGroupInfo.ClientSubnets'

# Check the subnet CIDRs
aws ec2 describe-subnets --subnet-ids <subnet-id-1> <subnet-id-2> <subnet-id-3> \
  --query 'Subnets[].[SubnetId,AvailabilityZone,CidrBlock]' --output table

The drawback is that other resources in the same subnets are also covered by the rule. If your security requirements are strict, dedicated subnets for MSK reduce this drawback. You can apply the same CIDR-based approach to devices other than security groups, such as on-premises firewalls.

2. If you must keep managing individual IPs, use a managed prefix list

Register the broker IPs in a customer-managed prefix list, and have your security group rules reference only that list (Optimize AWS infrastructure management with prefix lists). When an IP changes, you update the entries in the prefix list instead of editing rules across multiple security groups.

Note: You can't modify an existing rule to use a prefix list, so you must create a new rule. Also, a rule that references a customer-managed prefix list counts as the maximum size of the prefix list against your security group rule quota. Don't set the maximum size larger than you need.

With this approach, you still have to update the list when IPs change. Use the list-nodes output from the diagnosis step as the source for updates.

3. Have clients connect with the bootstrap string

Separately from access control, don't put broker IPs directly in client configurations. Clients connect to the broker host names in the bootstrap brokers string and then get the broker list from Kafka metadata. After a removal or expansion, run get-bootstrap-brokers to get the latest bootstrap string and update your client configurations. If a removed broker remains in the bootstrap string, the initial connection to that host can fail.

Note: The same principle applies if you registered broker IPs as ip targets in a Network Load Balancer target group to connect on custom ports. When broker IPs change or brokers are added, you must also update the target group registrations.

What to check before a removal or expansion

1. Before the operation: Save the list-nodes output. If you plan to keep brokers 1, 2, and 3, move the user topic partitions to those brokers first.
2. After the operation: Run list-nodes again to compare broker IDs and IPs, and update any rules that are based on IPs. Update clients with the latest bootstrap string.
3. Check security groups after expansion: If you ever changed the security group of the broker ENIs directly in the Amazon EC2 console, brokers added by expansion are associated with the original security group from when the cluster was created. All brokers must be associated with the same security group for the cluster to work correctly, so you must also assign that security group to the ENIs of the new brokers (Changing an Amazon MSK cluster's security group).

Related information

• Remove a broker from an Amazon MSK cluster
• Best practices for Apache Kafka clients
• Changing an Amazon MSK cluster's security group
• Security group rules
• Optimize AWS infrastructure management with prefix lists