Questions tagged with Amazon Kinesis

Content language: English

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

1
answers
0
votes
43
views
asked 7 months ago

V3 JS SDK Kinesis Client getting ERR_HTTP2_INVALID_SESSION error

Hi There, I am trying out Kinesis Client in [JS SDK V3](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-kinesis/globals.html). When I create a Kinesis Client in the global scope and reuse the same client for all further Kinesis ingestion, I am noticing that after a while I am getting the following error. ``` 1|new-sig | { Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed 1|new-sig | at ClientHttp2Session.request (internal/http2/core.js:1559:13) 1|new-sig | at Promise (/home/ec2-user/signaling-v7.temasys.io/node_modules/@aws-sdk/node-http-handler/dist-cjs/node-http2-handler.js:57:33) 1|new-sig | at new Promise (<anonymous>) 1|new-sig | at NodeHttp2Handler.handle (/home/ec2-user/signaling-v7.temasys.io/node_modules/@aws-sdk/node-http-handler/dist-cjs/node-http2-handler.js:37:16) 1|new-sig | at stack.resolve (/home/ec2-user/signaling-v7.temasys.io/node_modules/@aws-sdk/client-kinesis/dist-cjs/commands/PutRecordCommand.js:27:58) 1|new-sig | at /home/ec2-user/signaling-v7.temasys.io/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:5:32 1|new-sig | at /home/ec2-user/signaling-v7.temasys.io/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:11:26 1|new-sig | at process._tickCallback (internal/process/next_tick.js:68:7) '$metadata': { attempts: 1, totalRetryDelay: 0 } } ``` I am not getting an idea as to why this is happening. However when I use a custom requestHandler and disable Keep-Alive the client stops throwing the error (like the below code) ``` const { NodeHttpHandler } = require("@aws-sdk/node-http-handler"); const { Agent } = require("http"); const kinesisClient = new KinesisClient({ region: kinesisDataStream.region, requestHandler: new NodeHttpHandler({ httpAgent: new Agent({keepAlive: false}) })}); ``` Could you help me in understanding what's going on ? Thanks a lot.
1
answers
1
votes
78
views
asked 7 months ago

Why Records: [] is empty when i consume data from kinesis stream by python script?

i am trying to consume data using python script from kinesis data stream which is created successfully and data is produced or streamed to it successfully , but when running consumer script in python : ``` import boto3 import json from datetime import datetime import time my_stream_name = 'stream_name' kinesis_client = boto3.client('kinesis', region_name='us-east-1') response = kinesis_client.describe_stream(StreamName=my_stream_name) my_shard_id = response['StreamDescription']['Shards'][0]['ShardId'] shard_iterator = kinesis_client.get_shard_iterator(StreamName=my_stream_name, ShardId=my_shard_id, ShardIteratorType='LATEST') my_shard_iterator = shard_iterator['ShardIterator'] record_response = kinesis_client.get_records(ShardIterator=my_shard_iterator, Limit=2) while 'NextShardIterator' in record_response: record_response = kinesis_client.get_records(ShardIterator=record_response['NextShardIterator'], Limit=2) print(record_response) # wait for 5 seconds time.sleep(5) ``` But the output of the message data is empty ('Records': []): ``` {'Records': [], 'NextShardIterator': 'AAAAAAAAAAFFVFpvvveOquLUe7WO9nZAcYNQdcS6f6a+YGrrrjZo1gULcu/ZYxC7AB+xVlUhgL9UFPrQ22qmcQa6iIsmuKWl26buBk3utXlVqiGuDUYSgqMOtkp0Y7pJwa6N/I0fYfl2PLTXp5Qz8+5ZYuTW1KDt+PeSU3992bwgdOm7744cxcSnYFaQuHqfa0vLlaRBTOACVz4fwjggUBN01WdsoEjKmgtfNmuHSA7s9LLNzAapMg==', 'MillisBehindLatest': 0, 'ResponseMetadata': {'RequestId': 'e451dd27-c867-cf3d-be83-edbe95e9da9f', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'e451dd27-c867-cf3d-be83-edbe95e9da9f', 'x-amz-id-2': 'ClSlC3gRJuEqL9YJcHgC2N/TLSv56o+6406ki2+Zohnfo/erFVMDpPqkEWT+XAeeHXCdhYBbnOeZBPyesbXnVs45KQG78eRU', 'date': 'Thu, 14 Apr 2022 14:23:21 GMT', 'content-type': 'application/x-amz-json-1.1', 'content-length': '308'}, 'RetryAttempts': 0}} ```
2
answers
0
votes
177
views
asked 8 months ago

KDA Studio App keep throwing glue getFunction error, but I didn't use any glue function

I followed [this AWS blog post](https://aws.amazon.com/blogs/aws/introducing-amazon-kinesis-data-analytics-studio-quickly-interact-with-streaming-data-using-sql-python-or-scala/) to create KDA app, and change the output sink into s3 instead of data stream, everything is working, and I can get the result in s3. However in the KDA error logs, glue keep throwing getFunction error almost every second I run the deployed app, but I only use glue to define input/output schemas, didn't use any glue function, so I wonder where is it come form. Please help to take a look. ``` @logStream kinesis-analytics-log-stream @message {"locationInformation":"com.amazonaws.glue.catalog.metastore.GlueMetastoreClientDelegate.getFunction(GlueMetastoreClientDelegate.java:1915)","logger":"com.amazonaws.glue.catalog.metastore.GlueMetastoreClientDelegate","message":"software.amazon.kinesisanalytics.shaded.com.amazonaws.services.glue.model.EntityNotFoundException: Cannot find function. (Service: AWSGlue; Status Code: 400; Error Code: EntityNotFoundException; Request ID: <Request ID>; Proxy: null)","threadName":"Thread-20","applicationARN":<applicationARN>,"applicationVersionId":"1","messageSchemaVersion":"1","messageType":"ERROR"} @timestamp <timestamp> applicationARN <applicationARN> applicationVersionId 1 locationInformation com.amazonaws.glue.catalog.metastore.GlueMetastoreClientDelegate.getFunction(GlueMetastoreClientDelegate.java:1915) logger com.amazonaws.glue.catalog.metastore.GlueMetastoreClientDelegate message software.amazon.kinesisanalytics.shaded.com.amazonaws.services.glue.model.EntityNotFoundException: Cannot find function. (Service: AWSGlue; Status Code: 400; Error Code: EntityNotFoundException; Request ID:<Request ID>; Proxy: null) messageSchemaVersion 1 messageType ERROR threadName Thread-20 ``
0
answers
1
votes
31
views
asked 8 months ago

synchronous queue implementation on AWS

I have a queue in which producers are adding data and consumers wants to read and process it. In the diagram below producers are adding data in a queue with (Px, Tx, X) example (P3, T3,10) here, P3 is the producer ID, T3 is the number of packets required to process and 10 is data. for (P3, T3,10) consumer needs to read 3 packets from the P3 producer so In the Image below, one of the consumer needs to pick (P3, T3,10), (P3, T3,15) and (P3, T3,5) and perform a function on data that just add all the number that is 10+15+5 = 30 and save 30 to DB. Similarly there is a case for P1 producer (P1,T2,1) and (P1,T2,10) sum = 10+1 = 11 to DB. I have read about AWS Kinesis but it has issues, all consumers read the same data which doesn't fit my case. The major issue is how we can limit consumers for: 1 - Read data queue in synchronous. 2 - If one of the consumers has read (P1, T2,1) then only this consumer can read the next packet from the P1 producer (This point is the major issue for me as the consumer need to add those two number) 3 - This can also cause deadlock as some of the consumers will be forced to read data from a particular producer only because they have already read one packet from the same producer, now they have to wait for the next packet to perform add. I have also read about SQS and MQ but the above challenges still exist for them too. ![Image](https://i.stack.imgur.com/7b3Mm.png) [https://i.stack.imgur.com/7b3Mm.png](https://i.stack.imgur.com/7b3Mm.png) My current approach: for N produces I have started N EC2 instances, producers send data to EC2 through WebSocket (Websocket is not a requirement) and I can process it there easily. As you can see having N EC2 to process N producers will cause budget issues, how can I improve on this solution.
1
answers
0
votes
44
views
asked 8 months ago

AWS Go SDK not finding the credentials file at C:/###/.aws/credential.

I am using Amazon Kinesis and the [Go SDK for AWS](https://github.com/aws/aws-sdk-go), but I'm getting an error. This is my code: ```go package main import ( "math/rand" "strings" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" _kinesis "github.com/aws/aws-sdk-go/service/kinesis" ) func main() { session, err := session.NewSession(&aws.Config{ Region: aws.String("us-east-1"), }) handleErr(err) kinesis := _kinesis.New(session) laugh := strings.Builder{} laughingSounds := []string{"haha", "hoho", "hehe", "hehehe", "*snicker*"} for i := 0; i < 10; i++ { laugh.WriteString(laughingSounds[rand.Intn(len(laughingSounds))]) } _, err = kinesis.PutRecord(&_kinesis.PutRecordInput{ Data: []byte(laugh.String()), PartitionKey: aws.String("laughs"), StreamName: aws.String("laughs"), }) handleErr(err) } func handleErr(err error) { if err != nil { panic(err) } } ``` However, when I run this I get an error: ``` panic: UnrecognizedClientException: The security token included in the request is invalid. status code: 400, request id: dc139793-cd38-fb30-86a3-f92b6410e1c7 goroutine 1 [running]: main.handleErr(...) C:/Users/####/----/main.go:5 main.main() C:/Users/####/----/main.go:34 +0x3ac exit status 2 ``` I have run `aws configure`: ``` $ aws configure AWS Access Key ID [None]: #### AWS Secret Access Key [None]: #### Default region name [None]: us-east-1 Default output format [None]: ``` and the `C:/users/####/.aws/credentials` file is created with the correct configuration. But my program still wouldn't execute successfully. When that didn't work, I also set an environment variable like this: ``` $ $env:aws_access_key_id="####" ``` It still doesn't work. > Version info: ``` $ pwsh -v PowerShell 7.2.2 $ aws -v aws-cli/2.4.27 Python/3.8.8 Windows/10 exe/AMD64 prompt/off ``` OS: Windows 11 (version 21H2). Thanks in advance!
1
answers
0
votes
28
views
asked 8 months ago