Having trouble getting the Java Example Consumer to work / understanding Kinesis Data Streams concepts

0

So far, I read this article: https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html?icmpid=docs_console_unmapped

Now I am trying to implement a minimalistic example according to this article

https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-kinesis-stream.html

The following things are not clear to me:

What resource do I have to specify in the consumerARN? The Java Application is a Spring Boot Application running on AWS Fargate. So will the ARN be the ECS Task? How do I make the shardId deterministic? I have no control over the client that is putting the records on the data stream. Polling the listShards method continuously does not seem like a good idea. I also don't know how many records will be put on the stream in a given timeframe.

What exactly is the relation between the partitionKey and the shardId? Can different partitionKeys result in the same shardId? If the client would always use the same partitionKey for all records, will the shardId stay the same forever?

If there were multiple shards, should I handle each shard with a separate SubscribeToShardRequest on a new Thread? If yes, how many Threads should I create at max on an ECS instance with 2GB RAM and 1vCPU / how many shards can a a Kinesis Data Stream have at maximum?

gefragt vor 2 Jahren106 Aufrufe
Keine Antworten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen