Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Kinesis PutRecord Causing Lambda to Timing out

0

I have a Lambda Function written in Golangg, that sends data to Kinesis using the PutRecord function. The Lambda was able to process millions of data daily, but occasionally it was timing out a few times a day. We put debug lines on the Lambda log and found out that the Lambda hung when calling PutRecord to Kinesis, hence we got the Task timed out (we set it to 30 seconds, and the lambda p99.99 duration was around 700ms).

The timed-out lambda will retry and it did successfully run the second time, so we are sure that the data sent to Kinesis was not the issue here. Looking into Kinesis metrics we don't see anything suspicious, (not sure if we are looking at the right metrics).

So the question is what is happening to the Kinesis PutRecord? What metrics do we need to monitor to find out the cause? How do we solve the timed-out issue?

Topics
AnalyticsServerlessInternet of Things (IoT)Compute
Tags
Amazon KinesisAWS Lambda
Language
English
asked 2 years ago423 views
1 Answer
1

Things fail all the time and nothing is perfect.

If the vast majority of your Kinesis calls succeed in less than a second, why not set the network/TCP timeout on the Kinesis calls to (say) five seconds? You already know the retry will work so why not make it work faster?

You might also consider performing some sort of random exponential backoff. There's a great article about this in the Amazon Builder's Library - you haven't said that the trigger is for the Lambda function but you might find that using a queue (such as SQS) is helpful here too.

AWS
EXPERT
answered 2 years ago
EXPERT
reviewed 2 years ago
  • dark knight
    2 years ago

    would something like this work? and create the Kinesis using this session.

    sess := session.Must(session.NewSessionWithOptions(session.Options{
	Config: aws.Config{
		MaxRetries: aws.Int(3), // Retry up to three times
		HTTPClient: &http.Client{
			Timeout: time.Second * 1, // 1-second timeout
		},
	},
}))
  • Brettski-AWS EXPERT
    a year ago

    I don't know - sorry. I'm not a JS expert.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content