Hi everyone,
I have a first lambda than scans an ftp server for new files every 2 minutes.
Once a new file is detected, the lambda function will send its location path as a message in an SQS queue.
A second lambda will get triggered as soon as a message is in the queue.
It will receive this message containing the location of the file and download it from the server to its tmp folder, then transfer it from its tmp folder to an S3 bucket.
To avoid the first lambda to pass the same files locations multiple times, I make it first check the file is not already in the bucket and not in the queue neither.
The problem is that the using: sqs.receive_message() doesn't return anything in my first lambda.
I assume it is due to the visibility timeout.
So my question is: Is there a way i can peek in the queue for messages that have not yet been processed by the consumer (second lambda)?
Would setting the visibility timeout to 0 (if possible) do the trick? If I then call sqs.receive_message() in my first lambda (not the consumer then), will that remove the messages from the queue?
Many thanks
Antoine
Have you looked into AWS Step Functions yet? I think it might solve a lot of your problems regarding chaining Lambda functions together.