boto3 sqs receive_message MaxNumberOfMessages

0

Hi, we have a normal sqs (not FIFO), and trying to pull messages from it through python boto3 receive_message call specified here : https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.receive_message

message = sqs.receive_message( QueueUrl=queue_url, AttributeNames=[ 'SentTimestamp' ], MessageAttributeNames=[ 'All' ], VisibilityTimeout=120, MaxNumberOfMessages=MAX_RECEIVE_MESSAGES, WaitTimeSeconds=MAX_WAIT_TIME_SECS )

and having a few questions.

  • With understanding WaitTimeSeconds > 0 is called long polling. Even if we have around 10 messages in sqs, setting WaitTimeSeconds = 2 secs, MaxNumberOfMessages =10, we still get 1 message by calling receive_message. Is that expected behavior?
  • Based on developer document, MaxNumberOfMessages does not guarantee receiving the specified number of messages but just give the API a chance to scan through servers which containing messages. Am I understanding it correctly?
  • We program a loop to call the above api multiple times (times =10) and do get 1 message each time - total 10 messages. We wonder if such programming style would hit some sort of API limit (only allow certain times of receive_message call in a period). Is there such limit existing?
  • Will we be charged by each receive_message call? What is the amount of charge by each call?
  • Is it true only when the queue has > 5000 messages then we will see long polling returns more than 1 message?
  • the api allow us to specify VisibilityTimeout. However, if the SQS has a specified visibility timeout during creation. Which one will take affect, the one specified in API or the one specified during SQS creation?
  • where in real life, we will have a million message in the sqs at the very beginning. if we set WaitTimeSeconds = 2 secs, will the call return in 2 secs or possibility less than 2 secs in case the single long polling returns 10 messages within 1 sec?
  • our lambda is waken up by cloudwatch event every minute (shortest time can be specified by cloudwatch event rule), once waken up, it will run into a loop which keeps calling receive_message with MaxNumberOfMessages =10 until either the loop runs N times or received M number of messages. Any issue with this design?

Sorry for so many questions. Thank you!

asked a year ago767 views
1 Answer
0

The number of messages that you will get depends on the load on the queue, where there will be more load, you will see more messages in a batch.

You are charged for each API call, the price is $0.40 per 1 Million API calls. You can find more pricing information here.

When you use long polling, the time you specify is the max wait time for messages. If there are messages in the queue, the call will return immediately.

The visibility timeout that you specify when calling the API has precedence over the visibility timeout defined on the queue.

After saying all of that, why are you consuming the messages this way? Why not trigger the function from a queue and let Lambda handle all of it for you? See the doc.

profile pictureAWS
EXPERT
Uri
answered a year ago
  • Appreciate your answer. We are running this way instead of configured the queue triggering the lambda is because this is very busy system (Actually we did try that in our production system and ran out of lambda number of instance allowance). If there are thousands or millions of message in the queue, we are very easy to run out of lambda instance limit and we did experience that. Is this a valid concern? Any other suggestions?

  • Just ask for a limit increase. We increase that limit easily. Otherwise, the way you operate now, you have a single concurrency, which may not be enough to handle all the messages.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions