Kinesis Source Connector for Apache Spark Structured Streaming

0

What is the AWS recommended way for using Kinesis as a Source with Apache Spark Structured Streaming if running Spark jobs on EMR (not AWS Glue).

There is a third party open source connector library on github that hasn't been updated for several years and doesn't support Enhanced Fan Out (EFO).

Databricks also provide a Kinesis source connector but from what I can tell you can't use this outside of the Databricks ecosystem.

The Apache Spark project also has native support for Kinesis in the older DStreams based Spark Streaming library. They don't appear to have built the equivalent for the newer Structured Streaming library.

Is there a well supported option for a Kinesis source connector that I have missed?

AWS
feita há 2 anos122 visualizações
Sem respostas

Você não está conectado. Fazer login para postar uma resposta.

Uma boa resposta responde claramente à pergunta, dá feedback construtivo e incentiva o crescimento profissional de quem perguntou.

Diretrizes para responder a perguntas