Glue 5 with Kafka Writer

Question

In Glue 4 I can run a job using kafka as writer format:
```
.write \
    .format('kafka') \
    .option('kafka.bootstrap.servers', args["kafka_bootstrap_servers"]) \
    .option('kafka.security.protocol', 'SASL_SSL') \
    .option('kafka.sasl.mechanism', 'AWS_MSK_IAM') \
    .option('kafka.sasl.jaas.config', 'software.amazon.msk.auth.iam.IAMLoginModule required;') \
    .option('kafka.sasl.client.callback.handler.class', 'software.amazon.msk.auth.iam.IAMClientCallbackHandler') \
    .option('topic', args["target_topic"]) \
    .save()
```
Everything works fine adding the aws-msk-iam-auth jar to the job config.

After upgrading to Glue 5 the same job raise the following exception:

```
AnalysisException: Failed to find data source: kafka. Please deploy the application as per the deployment section of Structured Streaming + Kafka Integration Guide.
```
even if my job is not configured to expect a streaming application.

Over that while opening the job in thew aws console a warning message alerts me that:
```
Configure Glue 4.0 in your Job details when using application connections incompatible with Glue 5.0
```
I am not sure to understand what kind of application connection is incompatible with Glue 5.

Any suggestion?

Andrea Campolonghi · Answer

Anyone found a solution on this ? Writing to Kafka in glue5 seems impossible.

re:Post Agent · Answer

The issue you're experiencing with Glue 5 and Kafka as a writer format is likely due to changes in the way Glue 5 handles certain data source integrations, particularly for Kafka.

In Glue 5, some application connections, including the Kafka writer format you're using, may not be directly compatible. This is why you're seeing the warning message about incompatibility with Glue 5.0.

To resolve this issue, you have a few options:

1. Revert to Glue 4.0: If your job works fine in Glue 4, you can configure your job to use Glue 4.0 instead of Glue 5. This should allow your existing code to run without modifications.

2. Use a Glue connection: Instead of specifying Kafka options directly in your code, create a Glue Kafka connection to manage your connection credentials. You can then reference this connection in your job configuration by providing the `connectionName` as an "Additional network connection" in your job settings.

3. Update your Kafka integration: Glue 5 may require a different approach to integrating with Kafka. You might need to use a different library or method to write to Kafka that's compatible with Glue 5. This could involve using a different Spark Kafka integration or updating your dependencies.

4. Check for updated Glue 5 Kafka connectors: AWS may have released updated connectors or libraries specifically for use with Glue 5 and Kafka. Check the AWS documentation or reach out to AWS support for the most up-to-date information on Kafka integration with Glue 5.

Remember to ensure that all necessary IAM configurations are in place, especially if you're using Amazon MSK with IAM authentication. Also, make sure any required JAR files (like aws-msk-iam-auth) are properly included in your job configuration for Glue 5.

If none of these solutions work, you may need to consult with AWS support for a Glue 5-specific solution for Kafka integration, as there might be changes in the way Glue 5 handles external data sources that haven't been fully documented yet.
      
**Sources**  
[Kafka connections - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-kafka-home.html)  
[Streaming ETL jobs in AWS Glue - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/add-job-streaming.html)

Glue 5 with Kafka Writer

Add your answer

Relevant content