Why is my Lambda function batch size smaller than the configured batch size?

2 minute read
0

When processing Amazon Kinesis Data Streams, my AWS Lambda function's batch size is smaller than the batch size that I configured. Why is my Lambda function receiving fewer records per invocation than the batch size that I configured in my Amazon Kinesis event source?

Resolution

The batch size that you set when configuring a Kinesis event source determines the maximum batch size that your Lambda function can process. The batch size that your function processes when it's invoked by a Kinesis event source can be lower than the batch size that you configure.

Four values determine the batch size that your Lambda function processes when it's invoked by a Kinesis event source:

  • The maximum batch size limit that you set when configuring your Kinesis event source.
  • The number of records received from the GetRecords action that Lambda makes when polling your event source.
  • The number of records that can fit within the 6 MB Lambda invocation payload size limit.
    Note: A larger record size means that fewer records can fit in the payload.
  • The amount of data in your Kinesis Data Streams.
    Note: If the traffic on your Kinesis Data Streams is low, then the batch size that your function processes will be smaller.

To calculate the approximate batch size that your function will process, use the following formula:

6000 KB Lambda invocation payload size limit ÷ The size of an individual record in your batch (in KB) = Approximate number of records processed for each batch

For example, if each record in your batch is 64 KB, you can expect your function to process about 90 records per batch.


Related information

Using AWS Lambda with Amazon Kinesis

Working with streams

AWS OFFICIAL
AWS OFFICIALUpdated 3 years ago