Kinesis Source Connector for Apache Spark Structured Streaming

0

What is the AWS recommended way for using Kinesis as a Source with Apache Spark Structured Streaming if running Spark jobs on EMR (not AWS Glue).

There is a third party open source connector library on github that hasn't been updated for several years and doesn't support Enhanced Fan Out (EFO).

Databricks also provide a Kinesis source connector but from what I can tell you can't use this outside of the Databricks ecosystem.

The Apache Spark project also has native support for Kinesis in the older DStreams based Spark Streaming library. They don't appear to have built the equivalent for the newer Structured Streaming library.

Is there a well supported option for a Kinesis source connector that I have missed?

AWS
已提问 2 年前121 查看次数
没有答案

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则