Lambda Scheduler to pull Data from Kafka Topic

Question

Hi, I have kafka topic from which my Lambda function would be schedule daily to pull some events. What's max events it can pull in one go , may be my kafka topic would have 5-10million events.
Do i need to batch the record pulling in multiple service polling for or any way to do this in one request .

Answer

Instead of getting Kafka events the way you are proposing, have you looked at using a Kafka event trigger for lambda. Lambda will process messages as they arrive on the topic. You can specify a batch size which by default is 100 but can go up to 10000.

If you are using self-managed Kafka, take a look here - https://docs.aws.amazon.com/lambda/latest/dg/with-kafka.html

If you are using Amazon MSK, take a look here - https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html

Lambda Scheduler to pull Data from Kafka Topic

Relevant questions

java.nio.channels.UnresolvedAddressException when trying to create a topic in Amazon MSK

Kafka Lambda Trigger (EventSourceMapping) - Credentials in secret not refreshed

Lambda concurrency for Self managed Kafka event source

Set group.id for Kafka event source for lambda?

AWS MSK kafka-authorizer.log

IoT Events: Publish IoT Topic action with dynamic topic name