Kinesis Source Connector for Apache Spark Structured Streaming

0

What is the AWS recommended way for using Kinesis as a Source with Apache Spark Structured Streaming if running Spark jobs on EMR (not AWS Glue).

There is a third party open source connector library on github that hasn't been updated for several years and doesn't support Enhanced Fan Out (EFO).

Databricks also provide a Kinesis source connector but from what I can tell you can't use this outside of the Databricks ecosystem.

The Apache Spark project also has native support for Kinesis in the older DStreams based Spark Streaming library. They don't appear to have built the equivalent for the newer Structured Streaming library.

Is there a well supported option for a Kinesis source connector that I have missed?

AWS
gefragt vor 2 Jahren122 Aufrufe
Keine Antworten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen