How to connect Glue to MSK with IAM authentication?

Question

When setting up a Glue Connection for our MSK cluster configured woth IAM authentication, the broker disconnects the connection from Glue jobs using the connection with a message that says SASL failed. We have given ghe role used on the Glue job full permission. Is it currently possible to connect using IAM authentication?

Answer

Currently, AWS Glue does not integrate with Amazon MSK clusters configured to use AWS IAM access control.

You can connect to MSK clusters using IAM Access Control from other AWS services, such as [Amazon EMR](https://aws.amazon.com/emr/) and [Amazon Kinesis Data Analytics](https://aws.amazon.com/kinesis/data-analytics/). In these cases, [you configure the Kafka client to use MSK IAM authentication](https://github.com/aws/aws-msk-iam-auth#configuring-a-kafka-client-to-use-aws-iam).

Also, [AWS Lambda](https://aws.amazon.com/lambda/) has its own integration with MSK clusters using IAM Access Control, through an [Event Source Mapping](https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html). By taking this approach, the Kafka client is configured for you.

Answer

Hello,

I have published a blog which discuss about how to integrate AWS Glue ETL job with MSK serverless using IAM authentication. Hope it helps.

https://aws.amazon.com/blogs/big-data/securely-process-near-real-time-data-from-amazon-msk-serverless-using-an-aws-glue-streaming-etl-job-with-iam-authentication/

Answer

Hello,

Now AWS Glue supports to process data from Kafka using IAM authentication menthod. For this you need to create a connection glue and select IAM authentication menthod.

How to connect Glue to MSK with IAM authentication?

관련 콘텐츠