AWS MSK access with SASL/SCRAM authentication

Question

I have an AWS MKS cluster with SASL/SCRAM authentication; I am trying to access it from a python script running in EKS fargate.

consumer = KafkaConsumer(
        "test",
        bootstrap_servers=[f"kafka-1:9096", f"kafka-2:9096"],
        auto_offset_reset='earliest',
        enable_auto_commit=True,
        group_id="G1",
        security_protocol="SASL_SSL",
        sasl_mechanism="SCRAM-SHA-512",
        sasl_plain_username=secret["username"],
        sasl_plain_password=secret["password"],
        value_deserializer=lambda x: json.loads(x.decode('utf-8')))

But I am getting this error:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'kafka-1'. (_ssl.c:997)

I can access Kafka if I enable unauthenticated access, but the issue is only with SASL/SCRAM authentication.

If I understand this correctly, I need a provide a certificate for my Kafka-bootstrap servers. How can I provide an SSL certificate for my Kafka-bootstrap server? Do I have to create AWS Private Certificate Authority?

Answer

SSL certificate_verify_failed errors typically occur as a result of outdated Python default certificates or invalid root certificates which means it is not a valid SSL certificate.

Amazon MSK uses SASL/SCRAM sign-in credentials authentication and to set up the same for a cluster, you create a Secret resource in AWS Secrets Manager, and associate sign-in credentials with that secret.

Please refer the developer documentation on Setting up SASL/SCRAM authentication for an Amazon MSK cluster here: https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html#msk-password-tutorial

Incase of any issues during the setup please refer https://repost.aws/knowledge-center/msk-sasl-scram-issues.

AWS MSK access with SASL/SCRAM authentication

Relevant content