Why do I see metrics for GetRecords when I invoke Lambda ESM through Kinesis enhanced fan-out mode?

Short description

For standard Kinesis data streams, Lambda polls shards in your stream for records at a rate of once-per-second for each shard. For Kinesis EFO, Lambda uses an HTTP/2 connection to listen for records that are pushed from Kinesis. When records are available, Lambda invokes your function and waits for the result.

Resolution

To troubleshoot this issue, take the following actions:

Confirm your permissions

The following permissions are required to create a Lambda function and set the Kinesis activation with an EFO consumer:

kinesis:GetRecords
kinesis:GetShardIterator

These permissions are included in the AWSLambdaKinesisExecutionRole managed policy, which supports standard and EFO consumers. If the previous permissions are missing in the execution role, then the following error is shown: "An Error occurred when creating the trigger: Cannot access event source <Kinesis stream ARN>."

Confirm that your AWS Identity and Access Management (IAM) role has the following permissions: DescribeStreamSummary, ListShards, GetShardIterator, and GetRecords actions on your stream and SubscribeToShard on your consumer.

Confirm your event source mapping

When the Lambda event source mapping is configured with a Kinesis EFO stream consumer, the Lambda function can handle messages from the specified Kinesis stream. The Kinesis GetRecords Amazon CloudWatch metric isn't present for the successful Lambda invocations. This is expected behavior because Lambda ESM uses push-based enhanced fan-out consumer.

Confirm the CloudWatch metrics seen and reason for them

If the Lambda invocation fails, then the Lambda service tries to process the same batch multiple times (based on Maximum Retry Attempts configured). During these retry attempts, metrics for the GetRecords API call are shown. This is because the Lambda service polls the record from the shard for the failed Lambda invocation attempt. Polling the record is a GetRecords API call that results in a related metric data point.

Related information

Amazon Kinesis Data Streams dimensions and metrics