How do I troubleshoot authentication and permission issues when I use my Amazon MSK cluster with SASL/SCRAM authentication turned on?

Resolution

"ClusterAuthorizationException" error in broker logs

When you access your Amazon MSK cluster, you might receive the following error message: "org.apache.kafka.common.errors.ClusterAuthorizationException: Request Request(processor=11, connectionId=INTERNAL_IP-INTERNAL_IP-0, session=Session(User:ANONYMOUS,/INTERNAL_IP), listenerName=ListenerName(REPLICATION_SECURE), securityProtocol=SSL, buffer=null) is not authorized."

You receive the preceding error when both of the following conditions are true:

• Your Amazon MSK cluster has SASL/SCRAM authentication turned on.
• You set resourceType to CLUSTER and operation to CLUSTER_ACTION in the access control lists (ACLs) for your cluster.

The Amazon MSK cluster doesn't support the preceding settings because the settings prevent internal Apache Kafka replication. When both the preceding conditions are true, the identity of the brokers is ANONYMOUS for inter-broker communication. If you need your cluster to support these ACLs when you use the SASL/SCRAM authentication, then you must grant permissions for ALL operations to the ANONYMOUS user. This prevents the restriction of replication between the brokers.

To grant permission to the ANONYMOUS user, run the following command:

./kafka-acls.sh --authorizer-properties
zookeeper.connect=example-ZookeeperConnectString --add --allow-principal
User:ANONYMOUS --operation ALL --cluster

SASL authentication error

If you changed the username or password for your AWS Secrets Manager secret, but the old credentials are still active, you receive an authentication error.

A Secrets Manager secret can be associated with only a single user. When the user no longer needs access to the cluster, you must disassociate the secret and update the ACL.

When you update the username on the secret, Secrets Manager doesn't automatically disassociate the old username. It's a best practice to create a new secret for the new user. If you need to use the old secret, first disassociate the secret, and then update the credentials before you associate the secret again. For more information, see Working with users.

Non-admin users can create and modify ACLs without authorization

By default, all users have permission to create or modify ACLs. To prevent non-admin users from ACL modification through Apache ZooKeeper, place the ZooKeeper nodes in a separate security group. Also, make sure that all the Apache Kafka commands and client connections run through bootstrap brokers instead of Apache ZooKeeper.

Note: You can use the bootstrap string to perform all Apache Kafka operations.

Related information

Apache Kafka ACLs

Scram secrets