SQS "per request" billing way above actual usage...

Question

Hi,
I'm launching a project using 2 SQS queues. Looking at the billing page for this month, I see 1M requests in the free tier and 379,235 requests above (being charged). However I've had very little usage, juste a few testing clients. In fact, when I look at the monitoring tab of my 2 queues in the aws console for the past 3 months I only see a few hundreds of requests.
How come it be so different?
Thank you very much for your help,
Vincent

Accepted Answer

In SQS you pay for API requests, not messages. So sending a message to the queue is 1 API call (depending on the side of payload). Reading a message from the queue is another API call.

SQS is a pull based service. It has no way of notifying the consumer that there are messages in the queue. Therefore, the consumer must poll the queue to find if there are messages. As Lambda functions are event based, i.e., something needs to push the event to Lambda, and SQS is pull based, when you configure an SQS trigger for Lambda, the Lambda service (not the function), will constantly poll the queue, and when it finds messages, it invokes the function. It uses 5 pollers per queue, each using [long polling](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html) with a timeout of 20 seconds. If you will do the math it get to almost 1.3 million requests. ( 5 pollers * 3 reqs/min * 60 minutes/hour * 24 hours/day * 30 days/month * 2 queues = 1,296,000 requests). If you do the math, the price for Lambda polling an empty queue is ~$0.26/month. If there are messages, the polling interval shortens and the cost will be higher of course. When you have a large amount of messages, the number of pollers also increases to process the messages a fast as possible.

Even if you do not use Lambda, you probably have some other compute that polls the queue. The billed requests comes from these polling API calls.

Answer

And I'm adding my SQS billing report per API call
![SQS billing](/media/postImages/original/IMqf4H4b4uQyS_WyxcwzbFvA)

Answer

Thank you very much Uri for your prompt answer.
Indeed, I'm aware of the API calls counted as "requests" in the billing. But I wasn't aware the Lambda function was constantly polling! This does explain the bill!!
But it's crazy. And I can't really believe this, because 5 pollers querying every 20 seconds means that the lambda func would randomly take the messages into account after 0 to 4 seconds. However, I've never seen any latency between posting in the queue and my lambda func executing...

I'm using SAM to deploy my app and this configuration for my lambda function:
```
      Events:
        SqsTrigger:
          Type: SQS
          Properties:
            Queue: !GetAtt MyQueue.Arn
            BatchSize: 10

```
I thought this was "event based", meaning lambda func was being called when 1 to 10 messages would come in the queue...
Is there any way to change this (nb of pollers? interval?)? In my SAM config maybe?
At least, good news is this amount should be fixed per queue...
Thanks a lot again!

Answer

As I'm reading I'll self answer a few points
Polling starts at 5 concurrent pollers every 20 sec but it scales automatically, we can only set the max concurrency: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-params
I'm also seing in my queue monitoring the scaling effects and the number of empty responses:
![SQS monitoring](/media/postImages/original/IMUg5F71d1T52S5aLjbLzh2Q)
As I'm posting more messages I guess more lambda func are instanciated, each polling more.
So it's not a fix amount of calls per queue... :-/

SQS "per request" billing way above actual usage...

Relevant content