SQS batching S3 events issue

Hi Rodney,

Thanks for the response, the use case relies on files dropped in a bucket around a specific interval to be processed as a single batch/step in an EMR triggered by a lambda (1 producer S3 events, 1 consumer Lambda/EMR). We were relying on the Lambda batch capabilities to collect those events from SQS and invoke the EMR in a single step but that does not look possible judging from your response.

Hi MG,

I have tried both short and long polling without success, according to Rodney it cant accommodate batching of parellel events which is really a shame.

Ok yeah, now I remember :)

While the regular short polling returns immediately, even if the message queue being polled is empty, long polling doesn’t return a response until a message arrives in the message queue, or the long poll times out.

So basically it is impossible as Rodney wrote, you are right. It is because even you are using long polling, the message is returned immediately, the difference is only that the ReceiveMessage is checking all Queues in SQS Service and waiting for at least 1 message to return for a set time.

The thing is that the batch window is set to 300 seconds so it would be expected that it would pick all the messages from the Queue during that window but its not working like that unfortunately. Thanks MG for the help though :)

Ok, got it!

Lambda processes up to five batches at a time. This means that there are a maximum of five workers available to batch and process messages in parallel at any one time. Each worker shows a distinct Lambda invocation for its current batch of messages.

From here.

That's why it is spread between batches.

I dont believe its a lambda issue (even with concurrency set to one, batches arrive subsequently) as we tried polling through an EMR and parallel events arrived in seperate batches again.