MSK provides multiple security option. When to use what?

Hello,

You are correct, there are 4 ways to authenticate with an MSK cluster.

• Plaintext/Unauthenticated
• IAM
• SASL/SCRAM
• TLS with AWS Private Certificate Authority

However, choosing a preferred method to authenticate with your MSK cluster is based on your requirements and how your Kafka clients are configured. You need to look into how the Producer and Consumer applications you plan to use and identifying the best auth method you would like to proceed with with relation to the client.

Also note that you can select multiple authentication with in the cluster and use the different authentication for different clients. Each authentication method would require a different configuration when it comes to the producers and consumers. For example, you can use IAM auth with one producer and SASL/SCRAM for a consumer.

I will provide you with an overview of the different types of authentication methods to ensure which will best fit your use case.

1. Plaintext / Unauthenticated

No authentication is required for clients applications to produce and consume messaged. All actions are allowed.

2. IAM

IAM access control for Amazon MSK enables you to handle both authentication and authorization for your MSK cluster. This eliminates the need to use one mechanism for authentication and another for authorization.

For example, when a client tries to write to your cluster, Amazon MSK uses IAM to check whether that client is an authenticated identity and also whether it is authorized to produce to your cluster.

** NOTE: Please note that IAM only works with Java based clients. If your application is built with Python for example, IAM authentication will not be able to authenticate with MSK.

I do see you making use of connectors. If you plan to use MSK Connect, we only support Plaintext and IAM. If you have your own self-managed connectors, you will be able to authenticate with any of the 4 methods listed.

To start using MSK Connect, you will need IAM roles for MSK Connectors.

[+] Examples of IAM policies for MSK Connect: https://docs.aws.amazon.com/msk/latest/developerguide/mkc-iam-policy-examples.html

3. SASL/SCRAM

You can control authentication to your Amazon MSK clusters using sign-in credentials with usernames and passwords that are stored and secured using AWS Secrets Manager. Storing user credentials in Secrets Manager reduces the overhead of cluster authentication such as auditing, updating, and rotating credentials.

Please do take a look at the limitations of using SASL/SCRAM in the link below:

[+] Sign-in credentials authentication with AWS Secrets Manager - Limitations: https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html#msk-password-limitations

With this type of authentication method, you can use Kafka ACLs to restrict access to users [1].

To get started using SASL/SCRAM, you will need to setup AWS Secrets Manager [2]

4. TLS

You can enable client authentication with TLS for connections from your applications to your Amazon MSK brokers and ZooKeeper nodes.

To use client authentication, you need an AWS Private CA. A customer managed certificate will not work. The AWS Private CA can be either in the same AWS account as your cluster, or in a different account. To setup TLS authentication method, please see documentation below:

[+] Mutual TLS authentication: https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication.html

Note that choosing the preferred method of authentication for your MSK cluster will need to be based on your business logic and concept.

References:

[1] Apache Kafka ACLs: https://docs.aws.amazon.com/msk/latest/developerguide/msk-acls.html

[2] Sign-in credentials authentication with AWS Secrets Manager - How it works: https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html#msk-password-howitworks