How does data flow as you scale up Kinesis Data Steams and add KPU's to Kinesis Data Analytics.

0

I have an event processing architecture that's basically KPL Producer -> Kinesis Data Streams(Raw) -> KDA(Flink Aggregation) -> Kinesis Data Streams(Agg) -> Lambda -> Timestream . With a basic setup of 10 Raw Shards, 2 KPU's on the Flink App, and 5 Agg Shards(On Demand) and maybe 100 requests per second, my stack processes the data with maybe a delay of minute going through the whole workflow to Timestream. As I pre-scale things up for a much higher workload(that won't be arriving for several days), I notice that my current delay goes from 1 minute to 30 minutes to arrive in Timestream. The question I need to understand is where in my workflow is the latency being added. My current stack is now at 200 Raw Shards, 25 KPU's, and 15 Agg Shards(On Demand). I know the delay is most likely in Flink as I can see bursts of records being written out every 30 minutes using the incoming record count of my Agg KDS but my question is why? I know that it will clear up when my requests per second go thru the roof but I always wondered what underneath the hood is occurring in Flink to add this latency.

profile picture
질문됨 2년 전260회 조회
1개 답변
0
수락된 답변

Do you have any idle Kinesis shards? This would make the Apache Flink application watermark stall because it cant progress. This can be avoided by setting SHARD_IDLE_INTERVAL_MILLIS in the Kinesis Consumer config.

profile pictureAWS
flomair
답변함 2년 전
  • I'm sure I do. I see that this setting has been deprecated with the ".withIdleness" on the watermarking strategy. Thank you for pointing me in the right direction.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인