how to trigger a step function from a s3 object notification?
Can one directly invoke a step function directly from a s3 object notification ? for example , when object is created. also, does it work similar to lambda , when hooked up to the s3 object notification. for each object created , will it start a new instance of step function or can we make it such that all of the object notification goes to the same instance of step function, and as i understand step function are state machines , so say i have two states enabled and disabled. when object is created, state machine/step function will move to enable state , and for any other s3 object created notification that comes after the first one , as the step function is already in a enabled state , it won't do anything unless it is in a disabled state. once the files are processed it will move to disabled state.
To invoke a Step Function when an object is posted to S3, you'll need to create a Lambda function that starts an execution of the state machine. Each execution will be separate and independent for each uploaded object.
There isn't a concept of "enabled" and "disabled" for state machine executions - they are either running and in some state, or terminated.
You can find an experimental CDK solution construct that creates an S3 bucket connected to a Step Functions state machine here.
There are a few ways to go about that. The naive way would be to probe all the running executions, but the DescribeExecution API method is eventually consistent, and that could lead to errors. Instead, I would recommend keeping a list of object processing states in DynamoDB (using the S3 object ID as the primary key).
There is no direct integration between S3 and StepFunctions. You can send the S3 notification to a Lambda function, as suggest by @Michael_F, or a no code solution, by sending the event to EventBridge and setting a rule with a StepFunction target.
Saying that, I am not sure that StepFunctions is your right solution. It seems that you want to run some process on multiple files. If that is the case, you will not be able to "attach" into a running state machine. There may be some options using the Wait For Task Token integration pattern, but I am not sure it will solve what you are trying to do.
I think it would be best if you could explain your use case better.
Relevant questions
Conventions for Drawing Arrows in Architectural Diagrams
Accepted AnswerCan I specify GET URL path parameter in step function?
asked 3 months agoIntermittend InvalidSignature error when retrieving object from S3 Object Lambda accesspoint in lambda@edge function
asked 2 months agoAWS Lambda Function Triggers on S3 Event, But only Once in About 30 Mins
asked 2 years agoS3 object and lambda function in step machine with the asynchronous express workflows
Accepted Answerasked 4 months agohow to trigger a step function from a s3 object notification?
asked a month agoCoordinating Step functions from App Flow -> Event Bridge -> DynamoDB
asked 4 months agoS3: Configure object lock with replication (CRR) on aws s3 buckets
asked 24 days agoStep Function to Send Email on Error/Success
asked 4 months agoReturn Value from Lambda function triggered by SQS to individual client
Accepted Answerasked 4 months ago
@Michael_F - thanks. follow up question - if we need a lambda inbetween, it should be possible to check if state machine is already running/or in some state, right? instead of creating another instance of the state machine. and only start/run the same instance , if it is in terminated state.