Questions tagged with AWS Step Functions

Hi, I'm trying to create an AWS Steps Machine, I've got the majority of it working now, except I want to send an SES V2 email, this takes a JSON string variable called Template Data, where I want to send dynamic values, and I cannot work out how to do this from steps. Two things I've tried are; "TemplateData.$": "States.Format('{ \"name\":\"{}\", \"prizeValue\": \"{}\", \"emailAddress\": \"{}\"}', $.Query.Item.Name.S, $.PrizeValue , $.Query.Item.Email.S)" But as it uses {} as a variable placeholder, it won't let me use them elsewhere, I've tried escaping it to no avail. I've also tried using something akin to "TemplateData.$": "States.JsonToString({'Name.$' : 'Query.Item.Name'})" But I get "The value for the field 'TemplateData.$' must be a valid JSONPath or a valid intrinsic function call (at /States/Parallel/Branches[1]/States/SendEmail (1)/Parameters)"lg...

I am using a Step Function to execute a Glue Job. The Step Function is set to run in synchronous mode, however, there is usually a 2-4 minute lag from Glue Job completion to the point at which the Step Function considers the Glue Job complete and moves to the next step. For example, the Glue Job's last run took 15 minutes but the Step Function spent 19 minutes on this step. Has anyone else experienced this? Is my only option to execution in async mode and poll more often for completion?lg...

Here is a simple state machine to reproduce the bug: ``` { "StartAt": "First", "States": { "First": { "Type": "Choice", "Choices": [ { "Variable": "$.key", "NumericGreaterThanEqualsPath": "$.find", "Next": "ChoiceMatched" } ], "Default": "DefaultChoice" }, "ChoiceMatched": { "Type": "Pass", "Result": "ChoiceMatched", "End": true }, "DefaultChoice": { "Type": "Pass", "Result": "DefaultChoice", "End": true } } } ``` Here are the test cases: // Results in "DefaultChoice" for non-numeric value of key - Expected behavior ``` { "key": "ABC", "find": 0 } ``` // Results in "ChoiceMatched" when "key" is 0 and "find" is non-numeric - Incorrect behavior "ABC" was treated as numeric ``` { "key": 0, "find": "ABC" } ``` // Results in "ChoiceMatched" when "key" is 1 and "find" is non-numeric - Incorrect behavior was treated as numeric ``` { "key": 1, "find": "ABC" } ``` // Results in "DefaultChoice" when "key" is -1 and "find" is non-numeric - Incorrect behavior was treated as numeric ``` { "key": -1, "find": "ABC" } ``` The last three cases show that the "find" attribute's value ("ABC") was converted to numeric and precisely to a value of 0 when comparing against the "key" value of 0. This happens only when non-numeric is a Path variable and not the variable being compared (test case 1). Also, this coersion is only happening for numeric comparisons. Comparisons for other data types such as string etc. work as expected (there is another odd behavior with timestamps but I will ask a different question for that). The documentation for Choice step (https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-choice-state.html) clearly says "For each of these operators, the corresponding value must be of the appropriate type: string, number, Boolean, or timestamp. Step Functions doesn't attempt to match a numeric field to a string value." but this is clearly not the behavior. If this can't be fixed, at least the documentation must be updated to specify any conversion rules. Thanks.lg...

Hello, I have two SQS queues: one in us-east-1, and one in us-west-1. Currently my Step Function state machine is located in us-west-1, and has no trouble calling SQS API actions on the us-west-1 queue; however, any actions on the us-east-1 queue fail saying the queue cannot be found. Do Step Functions support using a queue outside of the state machine's region? If so, where is the parameter to set the queue region?lg...

Hello, I am working with AWS Step Functions and we are exploring the testing options. AWS Step Functions local seems like the way to go. I have tested it out locally as outlined in [this guide](https://aws.amazon.com/blogs/compute/mocking-service-integrations-with-aws-step-functions-local/), which works well. That requires using the AWS CLI to interact with SFN local. We are a Python shop so being able to continue using boto3 for interactions with AWS would be great. Is there anyway to use boto3 to query the local SFNs? I haven't seen any documentation about this and I've tried but the latest boto3 (1.24.71) doesn't seem to know about endpoints. Thankslg...

How do I choose between MWAA (Managed Workflows for Apache Airflow) and Step Functions? It seems like I can do the same thing with both. Is there a side-by-side code example showing the same use case in both systems so I can see the differences?lg...

Based on this [post](https://aws.amazon.com/blogs/developer/stepfunctions-fluent-api/), I was mulling over dynamically creating step functions state machines based on an in-house Domain Specific Language. The [quotas](https://docs.aws.amazon.com/step-functions/latest/dg/limits-overview.html) page says 'Maximum number of registered state machines' is 10,000. Does the term 'registered' mean that there are any other class of state machines that would not be subject to this quota? I'm trying to understand if there is any deeper meaning to the term registered in the context of state machine quotas. Thank you!lg...

I am building an ETL pipeline using primarily state machines, Athena, and the Glue catalog. In general things work in the following way: 1. A table, partitioned by "version", exists in the Glue Catalog. The table represents the output destination of some ETL process. 2. A step function (managed by some other process) executes "INSERT INTO" athena queries. The step function supplies a "version" that is used as part of the "INSERT INTO" query so that new data can be appended into the table defined in (1). The table contains all "versions" - it's a historical table that grows over time. My question is: What is a good way of exposing a view/table that allows someone (or something) to query only the latest "version" partition for a given historically partitioned table? I've looked into other table types AWS offers, including Governed tables and Iceberg tables. Each seems to have some incompatibility with our existing or planned future architecture: 1. Governed tables do not support writes via athena insert queries. Only Glue ETL/Spark seems to be supported at the moment. 2. Iceberg tables do not support Lake Formation data filters (which we'd like to use in the future to control data access) 3. Iceberg tables also seem to have poor performance. Anecdotally, it can take several seconds to insert a very small handful of rows to a given iceberg table. I'd worry about future performance when we want to insert a million rows. Any guidance would be appreciated!lg...

I have a use case wherein a lambda receives a REST request from the client. It then will need to call some N lambdas (scatter) using REST API and then wait to hear back from each of the N lambdas to collect (gather) the responses, consolidate the responses and respond back to the client. I'm thinking of using a step function (using parallel flow?) along with callback pattern when calling each of the N lambdas. Can you please share your ideas, blogs, github demos etc? Error handling is also essential.lg...

Hi Getting an error in the step function when trying to use the Put Bucket Website function. when placing a string "IndexDocument": "index.html" it does not allow me to save the function changes and states the error: `software.amazon.awssdk.services.s3.model.IndexDocument$BuilderImpl` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('index.html')]" I tried to place an argument - it allowed me to save - but the execution failed with a similar error: { "error": "States.Runtime", "cause": "An error occurred while executing the state 'Put Bucket Website' (entered at the event id #42). The Parameters '{\"WebsiteConfiguration\":{\"IndexDocument\":\"index.html\"},\"Bucket\":\"thetest1\"}' could not be used to start the Task: [Cannot construct instance of `software.amazon.awssdk.services.s3.model.IndexDocument$BuilderImpl` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('index.html')]" }lg...

I have a rest api gateway with direct integration with aws step functions. I want to get the email of the logged in user and pass it on to the step functions workflow. How can i get the email details in the integration rrequest? I have the following cloudformation but i cannot get the details from $context.authorizer.claims.email being passed on to the step functions workflow despite the other params being passed on. RequestTemplates: application/json: !Sub - |- #set($inputRoot = $input.path('$')) { "params": [ { "OrganisationName": $('$.OrganisationName') "OrgCreatedBy": "$context.authorizer.claims.email" } ] } { "input": "$params", "stateMachineArn": "${StateMachineArn}" } - { StateMachineArn: !Ref OrgAdminStateMachine }lg...

Hi Team, i am facing below error please help me for solution. "Error": "Lambda.TooManyRequestsException", "Cause": "Rate Exceeded. (Service: AWSLambda; Status Code: 429; Error Code: TooManyRequestsException; Request ID: ce268cb4-32b9-46ce-882e-0321506ef544; Proxy: null)" . i tried with MaxConcurrency: 200 and timeout in aws step function yml file stepFunctions: stateMachines: preProcessProduct: name: processProductARN-${sls:stage} definition: Comment: "Product updation/creation through the Amazon States Language using an AWS Lambda Function" StartAt: preProcessProduct States: preProcessProduct: Type: Task Resource: "arn:aws:lambda:${self:provider.region}:XXX:function:${self:service}-dev-preProcessProduct" ResultPath: $ Catch: - ErrorEquals: [States.ALL] ResultPath: $.preProcessProductError Next: userError Next: processProductMap processProductMap: Type: Map InputPath: $ # ItemsPath: $ MaxConcurrency: 200 Iterator: StartAt: processProduct States: processProduct: Type: Task Resource: "arn:aws:lambda:${self:provider.region}:XXX:function:${self:service}-dev-processProduct" ResultPath: $.outPut TimeoutSeconds: 60 Retry: - ErrorEquals: [ "TooManyRequestsException" ] IntervalSeconds: 60 MaxAttempts: 2 Catch: - ErrorEquals: [States.ALL] ResultPath: $.processProductError Next: processProductError End: truelg...