List queue messages yet unprocessed by consumer

Question

Hi everyone,

I have a first lambda than scans an ftp server for new files every 2 minutes.
Once a new file is detected, the lambda function will send its location path as a message in an SQS queue.

A second lambda will get triggered as soon as a message is in the queue.
It will receive this message containing the location of the file and download it from the server to its tmp folder, then transfer it from its tmp folder to an S3 bucket.

To avoid the first lambda to pass the same files locations multiple times, I make it first check the file is not already in the bucket and not in the queue neither.
The problem is that the using: sqs.receive_message() doesn't return anything in my first lambda.
I assume it is due to the visibility timeout.

So my question is: Is there a way i can peek in the queue for messages that have not yet been processed by the consumer (second lambda)?

Would setting the visibility timeout to 0 (if possible) do the trick? If I then call sqs.receive_message() in my first lambda (not the consumer then), will that remove the messages from the queue?

Many thanks

Antoine

Uri Segev · Answer

SQS does not have a way to peek messages in the queue. You should not follow this approach. If you want to make sure that each file is processed only once and that even the first function doesn't put the same file name twice into the queue, consider using SQS FIFO queues with [deduplication](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagededuplicationid-property.html).

List queue messages yet unprocessed by consumer

Add your answer

Relevant content