How do I troubleshoot errors when I use TLS client authentication to connect to my Amazon MSK cluster?

Resolution

If you activated client authentication on an Amazon MSK cluster, then make sure that you correctly configured the client with the SSL/TLS settings. Take the following actions for the error message that you have.

"Bootstrap broker" error message

If the client or producer can't connect to the cluster, then you receive an error message similar to the following:

"Bootstrap broker <broker-host>:9094 (id: -<broker-id> rack: null) disconnected"

To resolve, take the following actions:

• Make sure that your cluster has the SSL/TLS client authentication turned on and that the client configuration matches the cluster's authentication type.
  Note: If you turned on client authentication for your cluster, then you must include additional parameters for your AWS Private CA. For more information, see Mutual TLS client authentication for Amazon MSK.
• Check that the producer or consumer has the correct SSL/TLS client configuration to connect to an SSL/TLS encrypted cluster over port 9094.
• Make sure that you correctly configured the client.properties file and that it exists. The client.properties file must contain the following SSL configuration format:

security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks
ssl.keystore.location=/tmp/kafka.client.keystore.jks
ssl.keystore.password=your-password
ssl.key.password=your-password

"Key store could not be obtained" error message

If you incorrectly configured the truststore and tried to load the truststore files for the producer and consumer, then you receive the following error message:

"ERROR Modification time of key store could not be obtained. Failed to load keystore"

To resolve this issue, provide the correct path for the truststore file in the SSL/TLS configuration.

Example of an incorrect truststore path in the SSL configuration file

./kafka-console-consumer --bootstrap-server b-2.encryption.3a3zuy.c7.kafka.us-east-1.amazonaws.com:9094,b-1.encryption.3a3zuy.c7.kafka.us-east-1.amazonaws.com:9094 --topic test --consumer.config /home/ec2-user/ssl.config

"Handshake failed" error message

If you incorrectly configured the producer's or consumer's key store and an authentication failure occurs, then you receive one of the following error messages:

"Error when sending message handshake failed"

"Connection to node failed authentication due to: SSL handshake failed"

To resolve the error, make sure that you correctly configure the keystore.

Example of incorrect broker string for the producer's key store:

./kafka-console-producer --broker-list b-2.tlscluster.5818ll.c7.kafka.us-east-1.amazonaws.com:9094,b-1.tlscluster.5818ll.c7.kafka.us-east-1.amazonaws.com:9094,b-4.tlscluster.5818ll.c7.kafka.us-east-1.amazonaws.com:9094 --topic example --producer.config/home/ec2-user/ssl.config

Example of incorrect broker string for the consumer's key store:

./kafka-console-consumer --bootstrap-server b-2.tlscluster.5818ll.c7.kafka.us-east-1.amazonaws.com:9094,b-1.tlscluster.5818ll.c7.kafka.us-east-1.amazonaws.com:9094,b-4.tlscluster.5818ll.c7.kafka.us-east-1.amazonaws.com:9094 --topic example --consumer.config/home/ec2-user/ssl.config

"Keystore password was incorrect" error message

If the password for the key store or truststore is incorrect, then you receive an error message similar to the following:

"java.io.IOException: keystore password was incorrect"

To check whether the key store or truststore password is correct, run the following command:

keytool -list -keystore kafka.client.keystore.jksEnter keystore password:Keystore type: PKCS12Keystore provider: SUN
Your keystore contains 1 entry
schema-reg, Jan 15, 2020, PrivateKeyEntry,
Certificate fingerprint (SHA1): 4A:F3:2C:6A:5D:50:87:3A:37:6C:94:5E:05:22:5A:1A:D5:8B:95:ED

To view the output of the previous command, add the -v flag:

keytool -list -v -keystore kafka.client.keystore.jks

You can also run the preceding commands to check whether the key store is corrupted. If the AWS Secrets Manager secret key is incorrectly configured with the alias in the producer and consumer SSL/TLS settings, then you receive this error. To check, run the following command:

keytool -keypasswd -alias schema-reg -keystore kafka.client.keystore.jks

If your password for the alias's secret key is correct, then you're requested to enter a new password for the secret key.

If the key password for the alias is incorrect, then you receive an error similar to the following:

"keytool error: java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption."

To verify whether an alias is part of the key store, run the following command:

keytool -list -keystore kafka.client.keystore.jks -alias schema-reg

Example output:

Enter keystore password:schema-reg, Jan 15, 2020, PrivateKeyEntry,Certificate fingerprint (SHA1): 4A:F3:2C:6A:5D:50:87:3A:37:6C:94:5E:05:22:5A:1A:D5:8B:95:ED

Related information

Troubleshoot your Amazon MSK cluster

How do I troubleshoot connection errors for my Amazon MSK cluster?

How do I resolve errors when I use IAM or SASL/SCRAM client authentication to connect to my Amazon MSK cluster?