By using AWS re:Post, you agree to the Terms of Use

Questions tagged with Amazon Kinesis Data Streams

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

KCL1.x Java don't consuming records

I am using KCL 1.14.8 for Java. My Kinesis only have one shard, and I put a record into Kinesis every 30 seconds. For each record, the RecordProcessor will process it, processing each record may take several seconds(<10 seconds). This is worker configuration. ``` configuration.withInitialPositionInStream(InitialPositionInStream.LATEST) .withRegionName("eu-west-1") .withMaxRecords(10) .withIdleTimeBetweenReadsInMillis(1000L) .withCallProcessRecordsEvenForEmptyRecordList(false) .withRetryGetRecordsInSeconds(1) .withFailoverTimeMillis(60_000); ``` When the application start, It don't consuming records at all. the logs show this: ``` Aug 21, 2022 @ 21:24:14.943 Skipping shard sync due to the reason - Hash range is complete. Aug 21, 2022 @ 21:24:14.785 Number of pending leases to clean before the scan : 0 Aug 21, 2022 @ 21:24:00.144 Sleeping ... Aug 21, 2022 @ 21:24:00.144 Current stream shard assignments: shardId-000000000000 Aug 21, 2022 @ 21:23:14.911 Elected leaders: xxxxx-187a-4ceb-9714-e39ab1c7bb71 Aug 21, 2022 @ 21:23:14.785 Number of pending leases to clean before the scan : 0 Aug 21, 2022 @ 21:22:59.128 Sleeping ... Aug 21, 2022 @ 21:22:59.128 Current stream shard assignments: shardId-000000000000 Aug 21, 2022 @ 21:22:14.937 Skipping shard sync due to the reason - Hash range is complete. Aug 21, 2022 @ 21:22:14.785 Number of pending leases to clean before the scan : 0 Aug 21, 2022 @ 21:21:58.114 Current stream shard assignments: shardId-000000000000 Aug 21, 2022 @ 21:21:58.114 Sleeping ... Aug 21, 2022 @ 21:21:14.785 Number of pending leases to clean before the scan : 0 ``` please help.
0
answers
0
votes
17
views
asked a month ago

Kinesis data stream iterator age spikes

Hi, I am working on a system that uses kinesis to ingest data, this data is then processed by a lambda function and kinesis delivery stream. We have CloudWatch alarms set up that trigger if the iterator age for a stream goes above 10 seconds, as this means that lambda has crashed and can no longer process data, causing a build up of data. Over the last couple of weeks, we have seen an increase in alarm triggers. The cause is that the iterator age spikes to an insanely high number and then drops again within 1 to 2 minutes. There does not appear to be any pattern to the spikes, sometimes it is multiple a day, sometimes not for a couple of days. Some articles and post seem to mention that this could be caused by data built up or hitting a limit of the stream. However, when these spikes happen, there is usually no data being added to the stream. There have even been instances where only the stream was configured, but there was no system to put any data on the stream, and the spikes still occurred. This is what the spikes look like over a 3-day period. ![Enter image description here](https://repost.aws/media/postImages/original/IMNvNIagSNQs6N55IZCIODCw) I have already checked the following causes, and it is None of these issues: * Lambda stuck processing * Lambda error (there is also an alarm for this) * A lot of data added in a short time The stream is configured with 1 shard, enhanced fan-out is disabled, server-side encryption is enabled, the data retention is set to 1 day, and there are only 2 consumers. The kinesis delivery stream is configured with convert record format enabled, converting data to Parquet from JSON using standard AWS functions, there is no custom lambda. The delivery stream reports no failures. The lambda has starting position "LATEST" and max record age at "-1". What could be the cause of these spikes? Or how could I investigate this and figure out a solution?
1
answers
0
votes
39
views
asked 2 months ago