Having trouble getting the Java Example Consumer to work / understanding Kinesis Data Streams concepts

0

So far, I read this article: https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html?icmpid=docs_console_unmapped

Now I am trying to implement a minimalistic example according to this article

https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-kinesis-stream.html

The following things are not clear to me:

What resource do I have to specify in the consumerARN? The Java Application is a Spring Boot Application running on AWS Fargate. So will the ARN be the ECS Task? How do I make the shardId deterministic? I have no control over the client that is putting the records on the data stream. Polling the listShards method continuously does not seem like a good idea. I also don't know how many records will be put on the stream in a given timeframe.

What exactly is the relation between the partitionKey and the shardId? Can different partitionKeys result in the same shardId? If the client would always use the same partitionKey for all records, will the shardId stay the same forever?

If there were multiple shards, should I handle each shard with a separate SubscribeToShardRequest on a new Thread? If yes, how many Threads should I create at max on an ECS instance with 2GB RAM and 1vCPU / how many shards can a a Kinesis Data Stream have at maximum?

asked 2 years ago102 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions