跳至内容

Intermittent high latency (4+ minutes) in Kinesis GetRecords calls - How to handle timeouts effectively?

0

We're experiencing intermittent high latency issues with our Kinesis implementation:

Setup:

  • Kinesis stream with 64 shards
  • 40 consumers polling continuously
  • Using @aws-sdk/client-kinesis v3.74.0
  • Latency-sensitive application
  • both poller and consumers are in eu-west region

Issue: We've observed that some GetRecords calls are taking approximately 4 minutes to complete. This is happening intermittently and causing significant latency spikes in our processing.

Questions:

  1. Why would GetRecords calls take 4 minutes to complete sometimes? Is this a known issue?
  2. Is there a way to set a requestTimeout in the AWS SDK for Kinesis specifically? The documentation isn't clear about timeout configuration.
  3. If we implement our own timeout mechanism, would subsequent GetRecords calls be likely to succeed, or would they face the same issue?
  4. ** What's the recommended approach to handle such scenarios in latency-sensitive applications?**

Any insights into the cause and potential solutions would be greatly appreciated

已提问 2 年前167 查看次数

1 回答
0

Hello Samriddhi,

To address the latency issues, consider implementing detailed monitoring, sharding scaling, balancing, and record size monitoring. In Amazon Kinesis, each GetRecords request retrieves up to 1MB of data. If your records exceed this limit, multiple requests will be needed, increasing latency.

The following resources provide insights into mitigating latency issues in Kinesis:

已回答 2 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。