Troubleshooting Lambda Invocations Mismatch with SQS Queue Message Counts

Question

Hello AWS Community,

I'm experiencing an issue where the number of AWS Lambda function invocations is not aligning with the number of messages sent to my SQS queue. The invocation count is unexpectedly lower than the number of messages, which is concerning as message processing reliability is crucial for my application. I'm seeking insights on how to diagnose and fix this issue.

Here's a simplified version of my SAM template for context:
```yml
# SQS Queue Definition
MyQueue:
  Type: AWS::SQS::Queue
  Properties:
    QueueName: !Sub ${QueueName}Queue
    VisibilityTimeout: 800
    ReceiveMessageWaitTimeSeconds: 10
    DelaySeconds: 10
    RedrivePolicy:
      deadLetterTargetArn: !GetAtt DeadLetterQueue.Arn
      maxReceiveCount: 2

# Lambda Function Trigger
Events:
  MyEventTrigger:
    Type: SQS
    Properties:
      Enabled: true
      Queue: !GetAtt MyQueue.Arn
      BatchSize: 1
      FilterCriteria:
        Filters:
          - Pattern: '{"body":{"key1":[{"exists": false}]}}'
          - Pattern: '{"body":{"key2":[{"prefix": "https://example.com"}]}}'
```

The Lambda function is supposed to be triggered by new messages in the MyQueue. However, when I send multiple messages to the queue via an API Gateway from my terminal(with a loop that executes curl N times), the number of Lambda invocations is fewer than the messages sent.

Is this issue related to the SQS-Lambda integration, message deduplication, or a specific configuration in my SAM template? How can I ensure that each message in the queue reliably triggers a Lambda invocation?

Also here are some findings from my investigation.

- When there are 0 messages in the DLQ and MyQueue, after I call the gateway 5 times, no messages are in the DLQ
- There are 5 entries in the api gateway access logs
- From SQS monitoring dashboard, I see that 5 messages has been received and also delivered 5 times.
- This happens sometimes even with a single api call.
- One run of my function does not take longer than 7 seconds so I don't think it is related with the visibility timeout.

Your guidance would be greatly appreciated!

deniz · Accepted Answer

After further investigation, it is indeed related with my filter but not the one in my example. I also have another lambda function consuming messages from the same queue with a different filter. I believe this is what was happening:

Lambda Event Filter deletes messages from the Queue when the message do not match the filter criteria. If a queue has multiple lambda consumers with different event filters, there is a data loss when a filter which would match only one consumer was deleted by the other one because of a filter mismatch. It was hard to debug as there is no clear indication of who removed a message and why.

Uri · Answer

The most obvious suspect is the filter that you use. Could it be that not all message match the filter? If this is the case, messages that do not fit the filter are discarded and the function is not invoked.

Another, not so likely, suspect is the Delay param. When you send a message it will not be available for consumption for 10 seconds. Could it be that you did not wait enough time to see if the function is invoked?

Riku_Kobayashi · Answer

Hello.

How did you determine that Lambda has fewer invocations than SQS messages?  
If you look at the Lambda CloudWatch logs, are there any logs that were processed for SQS messages?

Troubleshooting Lambda Invocations Mismatch with SQS Queue Message Counts

Relevant content