polling from SQS containing 1 million messages without running out of lambda concurrency allowance.

Question

Hi,
  We have a sqs normal queue (not fifo queue) which the producer will publish thousands to millions message into, so the live message is constantly about 5k - 1million

we are using python boto3 receive_messaage to poll message. 
message = sqs.receive_message(
                QueueUrl=queue_url,
                AttributeNames=[
                    'SentTimestamp'
                ],
                MessageAttributeNames=[
                    'All'
                ],
                VisibilityTimeout=120,
                MaxNumberOfMessages=MAX_RECEIVE_MESSAGES,
                WaitTimeSeconds=MAX_WAIT_TIME_SECS
            )

To consume the message
* We had a lambda subscribe to the queue and start processing the messages,  the lambda does not call other API just poll messages, based on a certain values of certain field, for instance student_last_name="Smith" then publish/re-route to different SNS topics.   In this design pattern with millions message going on, won't we run out of lambda concurrent instance limit very soon?   We actually had another similar queue where a similar lambda takes about 4 seconds to finish (it is a more complicated lambda interact with other systems' APIs), and we quickly ran out of lambda instances.
* Another design is having the lambda triggered by cloudwatch event rule every 1 min,  the lambda will use the mentioned python receive_message call by setting MaxNumberOfMessages=10 (max allowance), and calling receive_message in a loop.   The loop only exit either runs a certain times or collection around 10000 messages then start processing these 10000 messages , finally route them to different sns topics.

both of these design seems not perfect for us.  the first one, has its risk and the second one only allows us to process every 1 min.   I wonder AWS architect and expert could provide further guidance?   Thank you very much.

Answer

You should use the SQS Lambda trigger to consume the messages from the queue. Concurrency will auto scale between 0 to 1000, based on number of messages in the queue. If you have other functions in the account and you are concerned that the function's concurrency will prevent other functions from running, you have 2 options:

1. Request a limit increase.
2. Use Reserved concurrency on the function to limit its maximum concurrency.

I would recommend the first option, unless you want to limit the concurrency because the function calls some downstream service which has limited resources.

polling from SQS containing 1 million messages without running out of lambda concurrency allowance.

Relevant content