List queue messages yet unprocessed by consumer

0

Hi everyone,

I have a first lambda than scans an ftp server for new files every 2 minutes. Once a new file is detected, the lambda function will send its location path as a message in an SQS queue.

A second lambda will get triggered as soon as a message is in the queue. It will receive this message containing the location of the file and download it from the server to its tmp folder, then transfer it from its tmp folder to an S3 bucket.

To avoid the first lambda to pass the same files locations multiple times, I make it first check the file is not already in the bucket and not in the queue neither. The problem is that the using: sqs.receive_message() doesn't return anything in my first lambda. I assume it is due to the visibility timeout.

So my question is: Is there a way i can peek in the queue for messages that have not yet been processed by the consumer (second lambda)?

Would setting the visibility timeout to 0 (if possible) do the trick? If I then call sqs.receive_message() in my first lambda (not the consumer then), will that remove the messages from the queue?

Many thanks

Antoine

  • Have you looked into AWS Step Functions yet? I think it might solve a lot of your problems regarding chaining Lambda functions together.

1 Answer
1

SQS does not have a way to peek messages in the queue. You should not follow this approach. If you want to make sure that each file is processed only once and that even the first function doesn't put the same file name twice into the queue, consider using SQS FIFO queues with deduplication.

profile picture
EXPERT
Uri
answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions