EMR Spark Job Fails to Connect to MSK with IAM Auth - Timeout Waiting for Node Assignment Error

Question

Hello,

I am running an Apache Spark job on Amazon EMR that needs to connect to an Amazon MSK cluster configured with IAM authentication. The EMR cluster has an IAM role with full MSK permissions, and I can successfully access MSK bootstrap brokers via telnet and using Python Kafka clients with the same permissions.

However, when running my Spark Structured Streaming job on EMR, it fails with the error:

```
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeTopics
```

My Spark submit command includes all the necessary Kafka and AWS MSK IAM authentication jars, specifically:

```
spark-sql-kafka-0-10_2.12-3.5.1.jar
kafka-clients-3.5.1.jar
spark-token-provider-kafka-0-10_2.12-3.5.6.jar
```

EMR Version : emr-7.2.0 
MSK Version : 3.6.0

The Spark streaming read is configured as follows:

```
python
spark.readStream.format("kafka") \
  .option("kafka.bootstrap.servers", "") \
  .option("subscribe", "my_topic") \
  .option("kafka.security.protocol", "SASL_SSL") \
  .option("kafka.sasl.mechanism", "AWS_MSK_IAM") \
  .option("kafka.sasl.jaas.config", "software.amazon.msk.auth.iam.IAMLoginModule required;") \
  .option("kafka.sasl.client.callback.handler.class", "software.amazon.msk.auth.iam.IAMClientCallbackHandler") \
  .load()
```

I have verified:
   
1.  EMR IAM role has required MSK permissions (Connect, DescribeCluster, DescribeTopic, etc.)
2.  Network connectivity to MSK brokers on port 9098 (SASL_SSL)
3.  Using compatible versions of Kafka client and IAM auth jars

I do NOT want to manually manage or distribute custom truststore files, as I expected the EMR JVM to trust MSK's default certificates automatically.

What could be the cause of the TimeoutException waiting for node assignment from Kafka when all connectivity checks pass and IAM permissions are verified?

Are there any best practices or additional configurations needed specifically on EMR or Spark to authenticate successfully with MSK using IAM?

Any guidance or examples of a working Spark + MSK IAM auth setup on EMR would be highly appreciated!

Thank you.

re:Post Agent · Answer

Based on your description, there are several potential causes for the timeout error when connecting from EMR Spark to MSK with IAM authentication.

First, ensure you've included the `aws-msk-iam-auth` library in your dependencies. This is a critical component when using IAM authentication with MSK. You mentioned several jars, but this specific one wasn't listed. You can include it in your Spark submit command with:

```
--packages software.amazon.msk:aws-msk-iam-auth:
```

Second, check your security group configurations. Even if you can connect via telnet, Spark executors might be running on different nodes with different security group rules. Ensure the security group attached to your EMR cluster allows outbound traffic to the MSK cluster, and the MSK cluster's security group allows inbound traffic from the EMR security group on port 9098.

Third, verify that your EMR application is configured with proper VPC access. When using the Kafka connector with IAM authentication for MSK, your EMR Serverless application must have VPC access configured. The subnets must have network connectivity to access the Kafka cluster and should have internet access through a NAT Gateway to download the Maven dependencies.

Fourth, double-check your Kafka configuration properties. The key properties for MSK IAM authentication should be:

```
kafka.security.protocol = SASL_SSL
kafka.sasl.mechanism = AWS_MSK_IAM
kafka.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required;
kafka.sasl.client.callback.handler.class = software.amazon.msk.auth.iam.IAMClientCallbackHandler
```

Finally, if you're using EMR Serverless, ensure your job execution role has the necessary permissions to interact with MSK. The role should have permissions like `kafka-cluster:Connect`, `kafka-cluster:DescribeCluster`, `kafka-cluster:ReadData`, and `kafka-cluster:DescribeTopic`.

If all these configurations are correct and you're still experiencing issues, consider enabling more detailed logging for the Kafka client to get more insights into the connection problems.  
**Sources**  
[Build a secure serverless streaming pipeline with Amazon MSK Serverless, Amazon EMR Serverless and IAM | AWS Big Data Blog](https://aws.amazon.com/blogs/big-data/build-a-secure-serverless-streaming-pipeline-with-amazon-msk-serverless-amazon-emr-serverless-and-iam/)  
[msk connector not able to fetchdata from msk cluster.. timeout error INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:23 | AWS re:Post](https://repost.aws/questions/QUmzCdTXUgSMSjlPj9qklklQ/msk-connector-not-able-to-fetchdata-from-msk-cluster-timeout-error-info-adminclient-clientid-adminclient-1-metadata-update-failed-org-apache-kafka-clients-admin-internals-adminmetadatamanager-23)  
[Supported streaming connectors - Amazon EMR](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/jobs-spark-streaming-connectors.html)  
[Connect to AWS MSK cluster from Local mac | AWS re:Post](https://repost.aws/questions/QUW0c-V5bSQYKB5iRNguIhiQ/connect-to-aws-msk-cluster-from-local-mac)

EMR Spark Job Fails to Connect to MSK with IAM Auth - Timeout Waiting for Node Assignment Error

Add your answer

Relevant content