Kinesis Source Connector for Apache Spark Structured Streaming

0

What is the AWS recommended way for using Kinesis as a Source with Apache Spark Structured Streaming if running Spark jobs on EMR (not AWS Glue).

There is a third party open source connector library on github that hasn't been updated for several years and doesn't support Enhanced Fan Out (EFO).

Databricks also provide a Kinesis source connector but from what I can tell you can't use this outside of the Databricks ecosystem.

The Apache Spark project also has native support for Kinesis in the older DStreams based Spark Streaming library. They don't appear to have built the equivalent for the newer Structured Streaming library.

Is there a well supported option for a Kinesis source connector that I have missed?

AWS
demandé il y a 2 ans118 vues
Aucune réponse

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions