Kinesis Source Connector for Apache Spark Structured Streaming

0

What is the AWS recommended way for using Kinesis as a Source with Apache Spark Structured Streaming if running Spark jobs on EMR (not AWS Glue).

There is a third party open source connector library on github that hasn't been updated for several years and doesn't support Enhanced Fan Out (EFO).

Databricks also provide a Kinesis source connector but from what I can tell you can't use this outside of the Databricks ecosystem.

The Apache Spark project also has native support for Kinesis in the older DStreams based Spark Streaming library. They don't appear to have built the equivalent for the newer Structured Streaming library.

Is there a well supported option for a Kinesis source connector that I have missed?

AWS
已提問 2 年前檢視次數 118 次
沒有答案

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南