By using AWS re:Post, you agree to the Terms of Use

Lambda Scheduler to pull Data from Kafka Topic

0

Hi, I have kafka topic from which my Lambda function would be schedule daily to pull some events. What's max events it can pull in one go , may be my kafka topic would have 5-10million events. Do i need to batch the record pulling in multiple service polling for or any way to do this in one request .

1 Answers
1

Instead of getting Kafka events the way you are proposing, have you looked at using a Kafka event trigger for lambda. Lambda will process messages as they arrive on the topic. You can specify a batch size which by default is 100 but can go up to 10000.

If you are using self-managed Kafka, take a look here - https://docs.aws.amazon.com/lambda/latest/dg/with-kafka.html

If you are using Amazon MSK, take a look here - https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html

profile picture
EXPERT
answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions