Durch die Nutzung von AWS re:Post stimmt du den AWS re:Post Nutzungsbedingungen
AWS re:Postre:Post

ProvisionedThroughputExceededException

0

Hello,

I am getting below exception when two glue jobs consumes same stream. The Stream has multiple shards. com.amazonaws.services.kinesis.model.ProvisionedThroughputExceededException: Rate exceeded for Shard - 826756392126/dev-vault-stream/shardId-000000000000 (Service: AmazonKinesis; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: dd6d1474-67a6-59c1-8094-f09703c5ef9c; Proxy: null)

My understanding is Kinesis can have a multiple consumers. Could you please help me here to understand the issue and how to fix it?

Thanks

Themen
AnalysenInternet der Dinge (IoT)
Tags
AWS GlueAmazon Kinesis DatenströmeAmazon Kinesis
Sprache
English
Asmita
gefragt vor 3 Monaten969 Aufrufe
1 Antwort
1

You can have multiple consumers but they share the same shard throughput of 2MB/s when provisioned.
You could add more shard to reduce the volume for each one but it's more robust to enable Fan-out reading, where readers don't compete for bandwidth.
You have to first enable it on the shard (or use on demand mode) and then configure the Glue 4 streaming job to use it:
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-kinesis-efo.html

profile pictureAWS
EXPERTE
Gonzalo Herreros
beantwortet vor 3 Monaten
  • Asmita
    vor 3 Monaten

    are there any drawbacks of implementing this approach?

  • Gonzalo Herreros EXPERTE
    vor 3 Monaten

    Obviously has a higher cost on the Kinesis side and it's a relatively new feature on the Glue side, so it's not as well tested as working on the default mode

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen

Relevanter Inhalt