Create Kafka Client Keystore certificate for connecting AWS MSK with mTLS Authentication from AWS Glue

Introduction:

Securing communication between AWS Glue and Amazon Managed Streaming for Apache Kafka (MSK) using mutual TLS (mTLS) authentication is a critical requirement for many enterprise applications. This guide walks through the step-by-step process of generating a Kafka client keystore (JKS) using AWS Certificate Manager Private Certificate Authority (ACM PCA) on an Amazon Linux EC2 instance. The resulting keystore enables secure, authenticated connections from AWS Glue jobs to your MSK cluster, ensuring data transmission security and compliance with security protocols.

Prerequisites:

• Amazon Linux EC2 instance or cloudshell
• AWS CLI with aws-pca configuration
• Java keytool utility
• Active AWS MSK cluster with ACM PCA configured

Steps:

1. Create the initial truststore by copying the Java cacerts:

cp /etc/pki/java/cacerts client.truststore.jks

2. Generate a new keystore with your client certificate:

keytool -genkey -keystore kafka.client.keystore.jks \
  -validity 100 \
  -storepass <your-store-password> \
  -keypass <your-key-password> \
  -dname "CN=<your-common-name>" \
  -alias kafkaprivateCA \
  -storetype pkcs12

3. Create a certificate signing request:

keytool -keystore kafka.client.keystore.jks \
  -certreq -file client-cert-sign-request \
  -alias kafkaprivateCA \
  -storepass <your-store-password> \
  -keypass <your-key-password>

4. Fix the certificate request format: The certificate should start with -----BEGIN CERTIFICATE REQUEST----- and end with -----END CERTIFICATE REQUEST-----

sudo sed -i -e 's!BEGIN NEW!BEGIN!g' client-cert-sign-request
sudo sed -i -e 's!END NEW!END!g' client-cert-sign-request

5. Issue the certificate using ACM PCA:

aws acm-pca issue-certificate \
  --certificate-authority-arn <YOUR_ACM_PCA_ARN> \
  --csr fileb://client-cert-sign-request \
  --signing-algorithm "SHA256WITHRSA" \
  --validity Value=100,Type="DAYS" \
  --region us-east-1

6. Retrieve the signed certificate:

aws acm-pca get-certificate \
  --certificate-authority-arn <YOUR_ACM_PCA_ARN> \
  --certificate-arn <YOUR_ACM_PCA_CERTIFICATE_ARN> \
  --region us-east-1

7. Create the signed certificate file:

   • Create a new file named signed-certificate-from-acm
   • Copy the Certificate string from the previous command's output
   • Append the CertificateChain string below it
   • Replace all \n with actual newlines

8. Import the signed certificate into the keystore:

keytool -keystore kafka.client.keystore.jks \
  -import -file signed-certificate-from-acm \
  -alias kafkaprivateCA \
  -storepass <your-store-password> \
  -keypass <your-key-password>

9. Verify the keystore contents:

keytool -list -v -keystore kafka.client.keystore.jks -storepass <your-store-password>

10. Configure the connection:
    • Upload kafka.client.keystore.jks to your S3 bucket.
    • Configure the certificate in your AWS Glue Kafka Connection > Choose TLS client authentication for Authentication method, pass the s3 location of kafka.client.keystore.jks in form field Kafka client keystore location and other details Kafka client keystore password and Kafka client key password

Note: Replace placeholder values (marked with < >) with your actual values before executing the commands.

Conclusion: By following these steps, you've successfully created and configured a Kafka client keystore for secure mTLS authentication between AWS Glue and MSK. This security implementation ensures that only authorized Glue jobs can connect to your MSK cluster, preventing unauthorized access and maintaining data security. Remember to store your keystore files securely, manage passwords carefully, and regularly rotate certificates according to your organization's security policies. The generated kafka.client.keystore.jks file, when properly configured in your AWS Glue Kafka Connection, establishes a secure foundation for your streaming data workflows.