Automatically invoke step functions in synchronously

0

I have to invoke a step function whenever there is a object created in an s3 bucket. the step function internally uses batch transform and I cannot use multiple batch transforms at the same time.

So I need to maintain a queue, such that a new step function starts only after the previous data from s3 has been processed

Any pointer

2 Answers
0

You can create a SQS FIFO queue between S3 events and your step functions (S3 Event->Lambda->SQS FIFO->Step Function). With FIFO queues you can guarantee the sequential execution of processing. Please note that you will need a Lambda as SQS FIFO does not support direct integration with S3 events.

AWS
answered 10 months ago
profile pictureAWS
EXPERT
reviewed 10 months ago
0

You have two main options:

  1. Make sure that you only have a single state machine invocation
  2. Allow for multiple, concurrent, invocations, but protect a specific state so that only one of the executions can actually get in (some sort of semaphore).

For the first option you should use EventBridge. All S3 events will be sent to EventBridge. The event will invoke a Lambda function. The function will check how many state machines are currently execution. If there is none, start a new one. If there is one, save the name of the object somewhere (DDB, SQS, etc.). When the state machine finishes, it emits an event to EventBridge. You will catch that event to check if there are any pending objects. If there are, start a new execution. All those operations need to be done in a way that are "thread safe", as you may have more than one event at the same time. You can use DDB conditional expressions for that.

The second option checks how many state machines are currently running the batch transform, If none, you perform the transform. If there is already one, you wait a little, and check again. In this case as well you can use DDB to increment a counter, but only if the value is 0. If it succeeded, you can get it. If it failed, you wait.

profile pictureAWS
EXPERT
Uri
answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions