Why does lambda consistently take 50 seconds to complete execution of invocations from event bridge scheduler whether the invocation count is 5000 or 100 or 10 it is taking 50 seconds

Question

I have done a poc to create a schedule job using AWS Event bridge schedule which connects to a lambda (rust runtime or python) to publish a message to kafak topic in MSK cluster
where each schedule should have a message that needs to be triggered based on the cron expression

I have 5000 schedules running every hour and 100 schedules running every 10mins
what I observed is whether I am running 5000 schedule or 100 schedules at the same time  **it is taking 50 seconds for lambda to complete**
The concurrent Executions is on an average of 60 when 5000 schedulers are triggering and 6 when 100 schedulers triggering the lambda
The throttling is zero in both the cases

even though the Full account concurrency is 1000 and Unreserved account concurrency is 1000 for the account
Service Quota of AWS Event Bridge Scheduler "Invocations throttle limit in transactions per second" is 1000

Why the time taken for completion of 10 or 100 or 5000 scheduler's triggering at the same time and publishing the message to Kafka is taking 50 seconds

it should be in less time for 100 schedulers right when compared to 5000 schedulers

I have tried provisioned concurrency in lambda as 10 and triggered 120 schedule
but I still see 50 seconds for the 120 schedules to get processed by lambda and as below
1.Total concurrent executions 5
2.Provisioned concurrent executions 5

from the lambda metrics the **init duration for rust runtime is average of 30ms and complete durations is average of 10ms and memory is 22MB**

Uri Segev · Answer

Are you scheduling all the 5000 schedules for the exact same time? Are you setting a FlexibleTimeWindow? If you are, this means that EventBridge will not trigger the events exactly at the time specified, but will distribute the invocations.

Why are you scheduling 5000 schedules for the same time? What are you doing there? Why not distribute them all over the hour?

re:Post Agent · Answer

Based on your observations, there are a few key factors that could explain why your Lambda functions consistently take 50 seconds to complete execution regardless of the number of invocations:

1. EventBridge Scheduler precision: EventBridge Scheduler does not provide second-level accuracy. It invokes targets with 60-second precision, which means there's inherent variability in when your functions actually start executing.

2. Lambda execution environment behavior: Each Lambda execution environment processes one event per invocation. When your Lambda function receives a batch of events, it processes them sequentially within that single environment. This means that even if you have high concurrency available, the actual processing might still be sequential within each Lambda instance.

3. Concurrency utilization: You mentioned that with 5000 schedulers, you're only seeing about 60 concurrent executions, and with 100 schedulers, you're seeing about 6. This suggests that your Lambda functions are scaling proportionally to the workload (roughly 1% of the total invocations are running concurrently), but they're not utilizing your full available concurrency of 1000.

4. Batch processing limitations: Even though your individual function execution time is very fast (10ms average duration), when processing multiple invocations, Lambda might be waiting for the batch window to complete before finalizing all executions. This could explain the consistent 50-second completion time regardless of batch size.

5. Post-processing phase: Lambda increases the concurrency counter when a function is invoked and decreases it only when the function returns a response. There's also a post-processing phase that counts toward total concurrency. This could be contributing to the consistent execution time.

To improve performance, you could:
- Consider implementing parallel processing within your Lambda functions to better utilize resources
- Adjust batch window settings if you're using batching
- Investigate if there are any bottlenecks in your MSK cluster that might be limiting throughput
- Monitor the PostRuntimeExtensionsDuration metric to see if extensions are causing delays

The fact that you're seeing similar execution times regardless of invocation count suggests there might be a fixed overhead or constraint in your processing pipeline that's independent of the number of events being processed.  
**Sources**  
[Why does my AWS Lambda function, triggered by an Event Bridge scheduler at 9:15 AM daily, consistently start processing 43 seconds late, despite similar functions executing almost instantly? | AWS re:Post](https://repost.aws/questions/QUScmzFuPnS2GhIfC494rKNA/why-does-my-aws-lambda-function-triggered-by-an-event-bridge-scheduler-at-9-15-am-daily-consistently-start-processing-43-seconds-late-despite-similar-functions-executing-almost-instantly)  
[SQS Lambda Batching | AWS re:Post](https://repost.aws/questions/QU2Ey8y85ISvyPMsty5YBS9g/sqs-lambda-batching)  
[Troubleshoot high concurrency on a Lambda event source | AWS re:Post](https://repost.aws/knowledge-center/lambda-high-concurrency)  
[Efficiently processing batched data using parallelization in AWS Lambda | AWS Compute Blog](https://aws.amazon.com/blogs/compute/efficiently-processing-batched-data-using-parallelization-in-aws-lambda/)

Why does lambda consistently take 50 seconds to complete execution of invocations from event bridge scheduler whether the invocation count is 5000 or 100 or 10 it is taking 50 seconds

Add your answer

Relevant content