- Newest
- Most votes
- Most comments
You can create a SQS FIFO queue between S3 events and your step functions (S3 Event->Lambda->SQS FIFO->Step Function). With FIFO queues you can guarantee the sequential execution of processing. Please note that you will need a Lambda as SQS FIFO does not support direct integration with S3 events.
You have two main options:
- Make sure that you only have a single state machine invocation
- Allow for multiple, concurrent, invocations, but protect a specific state so that only one of the executions can actually get in (some sort of semaphore).
For the first option you should use EventBridge. All S3 events will be sent to EventBridge. The event will invoke a Lambda function. The function will check how many state machines are currently execution. If there is none, start a new one. If there is one, save the name of the object somewhere (DDB, SQS, etc.). When the state machine finishes, it emits an event to EventBridge. You will catch that event to check if there are any pending objects. If there are, start a new execution. All those operations need to be done in a way that are "thread safe", as you may have more than one event at the same time. You can use DDB conditional expressions for that.
The second option checks how many state machines are currently running the batch transform, If none, you perform the transform. If there is already one, you wait a little, and check again. In this case as well you can use DDB to increment a counter, but only if the value is 0. If it succeeded, you can get it. If it failed, you wait.
Relevant content
- asked a year ago
- Accepted Answerasked a year ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated 7 months ago
SQS FIFO queues are not compatible with s3 events https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-compatibility.html
This is what I mentioned in my answer too. It does not mean that you cannot do it, you can use a Lambda function to receive the event and send it to SQS FIFO. Another way is to use Eventbridge : https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html
Thanks for the clarification. However, my batch function runs for more than 15 mins and lambda would timeout. Any alternate solution would be nice to know Thanks!
You can use the synchronous job execution feature for AWS Step function : https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-sync
So you can run your transform job in AWS Batch or ECS/EKS service and ask Step Function to wait for completion (this is not bound to 15 mins). There are some resources that can give you the idea :