SQS Lambda Batching

In your setup with EventBridge, SQS, and Lambda, the behavior of how messages are batched and processed by Lambda is governed by the configuration parameters you set for SQS and Lambda. Here’s a breakdown of how these settings work and why you’re observing the described behavior:

Configuration Details

1. SQS Batch Size: 100 messages
2. SQS Batch Window: 300 seconds (5 minutes)
3. Lambda Timeout: 5 minutes
4. SQS Visibility Timeout: 10 minutes

How SQS and Lambda Batching Works

• Batch Size: The maximum number of messages that Lambda will retrieve from the SQS queue in a single batch.
• Batch Window: The maximum amount of time that Lambda will wait before processing a batch of messages. This means Lambda will wait for either the batch size to be reached or the batch window time to elapse, whichever comes first.

Behavior Breakdown Scenario 1: 3 Events in the Queue

• Result: 3 separate Lambda invocations.
• Explanation: With a small number of events (less than the batch size of 100), Lambda will not wait for the batch window to elapse. It will invoke the Lambda function immediately for each message if the batch size is not met quickly, causing multiple invocations with smaller batches.

Scenario 2: 305 Events in the Queue

• First 200 Events:
• Result: 2 Lambda invocations, each processing 100 messages.
• Explanation: Since you have more than the batch size (100), Lambda will process the messages in batches of 100 as quickly as it can until the queue has fewer than 100 messages left.

Remaining 105 Events:

• Result: Processed after the batch window elapses.
• Explanation: The remaining 105 events are less than the batch size. Lambda waits for either the batch size to be met or the batch window to elapse. In this case, since no additional messages are arriving to meet the batch size of 100, Lambda waits for the entire batch window of 300 seconds before processing these messages.

Why the Remaining 105 Events Took 300 Seconds to Process

• Batch Window Behavior: When the number of messages in the queue is less than the batch size, Lambda waits for the batch window to complete. This is why the remaining 105 messages were processed after the 300-second batch window, causing the delay.

Recommendations to Avoid Delays

1. Adjust the Batch Window: If you want to reduce the latency for smaller batches, consider lowering the batch window. A shorter batch window will cause Lambda to invoke more frequently with smaller batches.

2. Adjust the Batch Size: If you frequently process small batches of messages, you might lower the batch size to ensure more frequent processing.

3. Monitor and Fine-Tune: Use CloudWatch metrics and logs to monitor the performance and tune the batch size and batch window settings according to your specific workload and latency requirements.

Example of Adjusted Configuration To process messages more quickly, you might set the batch window to a lower value, such as 30 seconds:

• Batch Size: 100 messages (keep this as it is)
• Batch Window: 30 seconds

This will ensure that even if the batch size is not met, messages will be processed within 30 seconds instead of waiting for the full 300 seconds.

By fine-tuning these parameters, you can achieve a balance between efficient batching and acceptable processing latency for your specific use case.