ProvisionedThroughputExceededException

0

Hello,

I am getting below exception when two glue jobs consumes same stream. The Stream has multiple shards. com.amazonaws.services.kinesis.model.ProvisionedThroughputExceededException: Rate exceeded for Shard - 826756392126/dev-vault-stream/shardId-000000000000 (Service: AmazonKinesis; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: dd6d1474-67a6-59c1-8094-f09703c5ef9c; Proxy: null)

My understanding is Kinesis can have a multiple consumers. Could you please help me here to understand the issue and how to fix it?

Thanks

Asmita
質問済み 3ヶ月前969ビュー
1回答
1

You can have multiple consumers but they share the same shard throughput of 2MB/s when provisioned.
You could add more shard to reduce the volume for each one but it's more robust to enable Fan-out reading, where readers don't compete for bandwidth.
You have to first enable it on the shard (or use on demand mode) and then configure the Glue 4 streaming job to use it:
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-kinesis-efo.html

profile pictureAWS
エキスパート
回答済み 3ヶ月前
  • are there any drawbacks of implementing this approach?

  • Obviously has a higher cost on the Kinesis side and it's a relatively new feature on the Glue side, so it's not as well tested as working on the default mode

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ