How do I resolve errors when I use IAM or SASL/SCRAM client authentication to connect to my Amazon MSK cluster?

Resolution

IAM client authentication errors

Failed authentication, access denied

If you use AWS Identity and Access Management (IAM) to authenticate, then you might receive the one of the following error messages:

• "Connection to node -1 (b-1.your-cluster.abc123.c2.kafka.us-east-1.amazonaws.com) failed authentication due to: Access denied"
• "org.apache.kafka.common.errors.SaslAuthenticationException: Access denied"

The preceding errors occur when access policies, permissions boundaries, and service control policies (SCPs) block users who don't pass the required authorization.

To resolve this issue, use IAM access control to make sure that the IAM role can perform cluster operations.

SaslAuthenticationException

You receive one of the following error messages:

• "org.apache.kafka.common.errors.SaslAuthenticationException: Too many connects"
• "org.apache.kafka.common.errors.SaslAuthenticationException: Internal error"

The preceding errors occur when you run your cluster on the kafka.t3.small broker type with IAM access control and you exceed the connection quota. The kafka.t3.small instance type accepts only one TCP connection for each broker per second. When you exceed the connection quota, your creation test fails. For more information, see How Amazon MSK works with IAM.

To resolve this issue, update the values for reconnect.backoff.ms and reconnect.backoff.max.ms to 1000 or higher in your Amazon MSK Connect worker configuration. Then, upgrade to a larger broker instance type, such as kafka.m5.large. For more information, see Right-size your cluster: Number of partitions per Standard broker.

SASL/SCRAM client authentication errors

Client SASL mechanism not turned on

You receive the one of the following error messages:

• "Connection to node -1 (b-1-testcluster.abc123.c7.kafka.us-east-1.amazonaws.com/3.11.111.123:9098) failed authentication due to: Client SASL mechanism 'SCRAM-SHA-512' not enabled in the server, enabled mechanisms are [AWS_MSK_IAM]"
• "Connection to node -1 (b-1-testcluster.abc123.c7.kafka.us-east-1.amazonaws.com/3.11.111.123:9096) failed authentication due to: Client SASL mechanism 'AWS_MSK_IAM' not enabled in the server, enabled mechanisms are [SCRAM-SHA-512]"

The preceding errors occur when the port number doesn't match the Authentication and Security Layer/Salted Challenge Response Authentication Mechanism (SASL/SCRAM) mechanism in the client properties file in the command to run cluster operations.

To communicate with brokers in a cluster that uses SASL authentication, use port 9096 for access from within AWS and port 9196 for public access.

To communicate with brokers that use IAM access control, use ports 9098 for access from within AWS and port 9198 for public access.

SASL credential authentication error

You receive the following error message:

"Connection to node -1 (b-3.testcluster.abc123.c2.kafka.us-east-1.amazonaws.com/10.11.111.123:9096) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512"

The preceding error occurs because the credentials that you used aren't valid. Make sure that you stored the user credentials in AWS Secrets Manager and associated the credentials with the MSK cluster.

When you access the cluster over port 9096, the user and password in Secrets Manager must be the same as the client properties.

When you run the get-secret-value command to retrieve the secrets, make sure that the password in Secrets Manager doesn't contain special characters.

ClusterAuthorizationException

You receive the following error message:

"org.apache.kafka.common.errors.ClusterAuthorizationException: Request Request(processor=11, connectionId=INTERNAL_IP-INTERNAL_IP-0, session=Session(User:ANONYMOUS,/INTERNAL_IP), listenerName=ListenerName(REPLICATION_SECURE), securityProtocol=SSL, buffer=null) is not authorized"

The preceding error occurs when you turn on SASL/SCRAM authentication for your MSK cluster and set resourceType to CLUSTER and operation to CLUSTER_ACTION in your cluster's access control lists (ACLs).

The MSK cluster doesn't support the preceding settings because the settings prevent the internal Apache Kafka replication. The brokers' identities appear as ANONYMOUS for inter-broker communication. If your cluster must support the ACLs and use the SASL/SCRAM authentication, then allow the ANONYMOUS user to use the ALL operation.

For ZooKeeper clusters, run the following command to grant the ALL operation to the ANONYMOUS user:

./kafka-acls.sh --authorizer-properties zookeeper.connect=example-ZookeeperConnectString --add --allow-principal User:ANONYMOUS --operation ALL --cluster

For KRaft mode clusters, run the following command to grant the ALL operation to the ANONYMOUS user:

./kafka-acls.sh --bootstrap-server BootstrapBrokerString --command-config client.properties --add --allow-principal User:ANONYMOUS --operation All --cluster

Related information

How do I troubleshoot errors I receive when I try to connect to my Amazon MSK cluster?