- Newest
- Most votes
- Most comments
Lambda concurrency with MSK is described here. As you can see, we start with concurrency of 1 and we increase it when the load is higher. To preserve order, we do not go above the number of partitions. This means that if you want to process the messages quicker, you will need to increase the number of partitions in your stream.
You could also increase the batch size and handle the messages in multiple threads, but that may break the order of messages.
Hi Bob, I hope you are doing great today. I understand that you are sending over 7000 messages to an MSK serverless cluster which are then processed by AWS Lambda. Your problem is that it takes up to 3 hours and the concurrency doesn't go above 3.
One of the things you can try is to increase the batch size and batch window from your trigger configuration.
A smaller batch size and batch window will cause an increase in the amount of requests going to the lambda. Keep in mind If you go over the default quota for your account, you might need to request the increase or adjust the trigger configuration to get a proper processing time/concurrency that make sense for your business.
Please, take a look in the following blog post for MSK and Lambda trigger [1] configuration and explanation.
I hope this information is helpful.
Have a great week!
Introducing AWS Lambda batching controls for message broker services
Relevant content
- asked 4 years ago
AWS OFFICIALUpdated 5 months ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated a year ago
