Questions tagged with AWS Step Functions

Hi Team, i am facing below error please help me for solution. "Error": "Lambda.TooManyRequestsException", "Cause": "Rate Exceeded. (Service: AWSLambda; Status Code: 429; Error Code: TooManyRequestsException; Request ID: ce268cb4-32b9-46ce-882e-0321506ef544; Proxy: null)" . i tried with MaxConcurrency: 200 and timeout in aws step function yml file stepFunctions: stateMachines: preProcessProduct: name: processProductARN-${sls:stage} definition: Comment: "Product updation/creation through the Amazon States Language using an AWS Lambda Function" StartAt: preProcessProduct States: preProcessProduct: Type: Task Resource: "arn:aws:lambda:${self:provider.region}:XXX:function:${self:service}-dev-preProcessProduct" ResultPath: $ Catch: - ErrorEquals: [States.ALL] ResultPath: $.preProcessProductError Next: userError Next: processProductMap processProductMap: Type: Map InputPath: $ # ItemsPath: $ MaxConcurrency: 200 Iterator: StartAt: processProduct States: processProduct: Type: Task Resource: "arn:aws:lambda:${self:provider.region}:XXX:function:${self:service}-dev-processProduct" ResultPath: $.outPut TimeoutSeconds: 60 Retry: - ErrorEquals: [ "TooManyRequestsException" ] IntervalSeconds: 60 MaxAttempts: 2 Catch: - ErrorEquals: [States.ALL] ResultPath: $.processProductError Next: processProductError End: truelg...

IS there any syntax documentation for how to run Step Functions with AWS CLI, and specifically for how to start an execution. I input the ARN, and the json input but keep getting an error that the --input argument is only expecting one argument. I was hoping someone could provide an example of the proper syntax for these parameters. Thanks!lg...

I know Step Functions use AWS SDK and it gets updated regularly so my questions: 1. What is the SDK update cycle for Step Functions? 2. Do we know if EMR Serverless can be integrated without using Lambda or other compute in between? Thanks Atharvalg...

Hello, we are trying to use `stepfunctions-local` (Docker version) and _Mocked Service Integrations_ to unit test our state machines. So far, it works great. The problem is, that the tests take too long, when there is a `Wait` or `Retry` logic in the step machine. `stepfunctions-local` execution waits the time that is defined in the state machine definition. I've seen that `stepfunctions-local` has the (rather undocumented) parameter `WAIT_TIME_SCALE`. From the Docs: https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local-config-options.html  I've set it to `0` but the execution still waits the seconds we specified in the definition, which makes the unit test unnecessarily slow. Can you help me understand wether I'm doing something wrong? What is the `WAIT_TIME_SCALE` parameter anyway? Thanks! Eneslg...

I updated my state machine to use a task token and now it's timing out when it gets to the choice. This was previously working, and the cloudwatch logs for the first supportticketparse lambda function aren't indicating that anything is wrong. My guess is that I have and error in the payload definition, but I'm not sure. Is anything obviously wrong with the definition? If not, where else can I look for a potential problems? ``` { "Comment": "A description of my state machine", "StartAt": "zen sort", "TimeoutSeconds": 60, "States": { "zen sort": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken", "OutputPath": "$.Payload", "Parameters": { "FunctionName": "arn:aws:lambda:us-east-1:363540650061:function:support-ticket-parse:$LATEST", "Payload": { "ExecutionContext.$": "$$", "APIGatewayEndpoint": "https://9084s1lza7.execute-api.us-east-1.amazonaws.com/test/", "Payload.$": "$" } }, "Next": "Choice" }, "Choice": { "Type": "Choice", "Choices": [ { "Variable": "$.name", "StringMatches": "New User Account ", "Next": "send email" }, { "Not": { "Variable": "$.name", "StringMatches": "New User Account " }, "Next": "Fail" } ], "Default": "Fail" }, "send email": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken", "OutputPath": "$.Payload", "Parameters": { "FunctionName": "arn:aws:lambda:us-east-1:363540650061:function:send_email:$LATEST", "Payload": { "ExecutionContext.$": "$$", "APIGatewayEndpoint": "https://9084s1lza7.execute-api.us-east-1.amazonaws.com/test/", "Payload.$": "$" } }, "Retry": [ { "ErrorEquals": [ "Lambda.ServiceException", "Lambda.AWSLambdaException", "Lambda.SdkClientException" ], "IntervalSeconds": 2, "MaxAttempts": 6, "BackoffRate": 2 } ], "End": true }, "Fail": { "Type": "Fail" } } } ```lg...

I am trying to put an item at dynamodb via step function. I am having issue when the item includes a map field which contains a list. For example. My step function ``` { "Comment": "A description of my state machine", "StartAt": "DynamoDB PutItem", "States": { "DynamoDB PutItem": { "Type": "Task", "Resource": "arn:aws:states:::dynamodb:putItem", "Parameters": { "TableName": "test", "Item": { "ID": { "S.$": "$.input" }, "Info": { "M.$": "$.info" } } }, "End": true } } } ``` My input to the step function ``` { "input": "Second", "info": { "name": "John", "age": "30", "cars": [ "Ford", "BMW", "Fiat" ] } } ``` The error I am recieving. An error occurred while executing the state 'DynamoDB PutItem' (entered at the event id #2). The Parameters '{"TableName":"test","Item":{"ID":{"S":"3"},"Info":{"M":{"name":"John","age":"30","cars":["apple","pear"]}}}}' could not be used to start the Task: [Cannot deserialize value of type `com.amazonaws.services.dynamodbv2.model.AttributeValue` from Array value (token `JsonToken.START_ARRAY`)]lg...

I am trying to invoke a spark script that runs over EMR serverless from a Step function. Nevertheless, I have not found an out of the box integration. One option is to use a lambda function that submit the spark job through Boto3, however, this solution does not allow the step function waits until the end of the spark job. so, how can I submit the spark job and block the step function until the job finalize?lg...

Hi, I have nested workflow for Step Functions where the outer state machine awaits for the inner state machine to return task token. Definitions: Outer State machine: ``` { "Comment": "A description of my state machine", "StartAt": "ChildProcessing", "States": { "ChildProcessing": { "Type": "Task", "Resource": "arn:aws:states:::states:startExecution.waitForTaskToken", "Parameters": { "StateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:child", "Input": { "parentTaskToken.$": "$$.Task.Token", "AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.$": "$$.Execution.Id" } }, "ResultPath": "$.output", "Next": "PostJobSuccess" }, "PostJobFailure": { "Type": "Task", "Resource": "arn:aws:states:::sqs:sendMessage", "Parameters": { "MessageBody.$": "$$.Execution.Input", "QueueUrl": "http://queueurl" }, "End": true }, "PostJobSuccess": { "Type": "Task", "Resource": "arn:aws:states:::sqs:sendMessage", "Parameters": { "MessageBody.$": "$$.Execution.Input", "QueueUrl": "http://queueurl" }, "End": true } } } ``` Inner State machine: ``` { "Comment": "Child processing.", "StartAt": "GetJobStatus", "States": { "GetJobStatus": { "Type": "Task", "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken", "TimeoutSeconds": 120, "Parameters": { "QueueUrl": "http://childqueueurl", "MessageAttributes": { "jobName": { "DataType": "String", "StringValue": "AsyncProcessingJob" } }, "MessageBody": { "taskToken.$": "$$.Task.Token", "machineExecutionId.$": "$$.Execution.Id" } }, "ResultSelector": { "input.$": "$$.Execution.Input.input", "parentTaskToken.$": "$$.Execution.Input.parentTaskToken" }, "Next": "CheckJobStatus" }, "CheckJobStatus": { "Type": "Choice", "Choices": [ { "Variable": "$.jobStatus", "StringMatches": "finished", "Next": "SendTaskSuccess" }, { "Variable": "$.jobStatus", "StringMatches": "failed", "Next": "SendTaskFailure" } ] }, "SendTaskFailure": { "Type": "Task", "Parameters": { "TaskToken.$": "$$.Execution.Input.parentTaskToken" }, "Resource": "arn:aws:states:::aws-sdk:sfn:sendTaskFailure", "End": true }, "SendTaskSuccess": { "Type": "Task", "Parameters": { "Output.$": "$.input", "TaskToken.$": "$$.Execution.Input.parentTaskToken" }, "Resource": "arn:aws:states:::aws-sdk:sfn:sendTaskSuccess", "End": true } } } ``` Here is my `aws-stepfunctions-local-credentials.txt` with fake credentials and setting STEP_FUNCTIONS_ENDPOINT to http://localhost:8083 ``` AWS_DEFAULT_REGION=us-east-1 AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_SESSION_TOKEN=test STEPFUNCTIONS_PORT=8083 STEP_FUNCTIONS_ENDPOINT=http://localhost:8083 STEPFUNCTIONS_HOST=stepfunctions-local ``` While executing this nested workflow, I am getting the error: ``` {"Type":"TaskSubmitFailed","PreviousEventId":22,"TaskSubmitFailedEventDetails":{"ResourceType":"aws-sdk","Resource":"sfn:sendTaskSuccess","Error":"Sfn.SfnException","Cause":"software.amazon.awssdk.services.sfn.model.SfnException: The security token included in the request is invalid. (Service: Sfn, Status Code: 400, Request ID: 864f4abc-6e26-4ce7-9cfa-63ade13dd6ca)"}} ``` Has anyone come across similar issue and found solution?lg...

I have a step function with a Map state. The Map state processes an items array which is the output of the task preceding the Map state. Sometimes the task preceding the Map state outputs an array of objects and everything works perfectly. Other times the task preceding the Map state outputs a single object (and not array) so the Map state cannot process it. How do I specify an OutputPath for the task preceding Map state which always returns an array of the tasks output even if the array consists of only a single object? ``` { "Comment": "A description of my state machine", "StartAt": "Lambda Invoke", "States": { "Lambda Invoke": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "Payload.$": "$", "FunctionName": "arn:aws:lambda:someregion:someuser:function:somefunctionname:$LATEST" }, "Next": "Map" }, "Map": { "Type": "Map", "End": true, "Iterator": { "StartAt": "Success", "States": { "Success": { "Type": "Succeed" } } } } } } ```lg...

Source: https://github.com/aws-samples/amazon-transcribe-post-call-analytics Found the AWS solution in the source link. By default, its behavior is configured through the inputs in the stack before you deploy it. It takes audio files uploaded to an input bucket as input to a lambda function that starts a step-function workflow. The confusing part is that it currently has a lot of variables in the [config file](https://github.com/aws-samples/amazon-transcribe-post-call-analytics/blob/develop/pca-server/src/pca/pcaconfiguration.py) which stores them in ssm. Wondering how I should go about making the input and behavior of the application more dynamic? Instead of uploading a file to trigger the workflow, have a normal lambda that points to a location for a file. Or specify what transcribe options are used etc. Do any of the configs need to be dynamically accepted as inputs?lg...

Is there any way to migrate step function/machine from one account to another?lg...

1. I am trying to implement a solution that integrates aws comprehend targeted sentiment along with step functions. And then make it public for people to use it as an api. 2. I need to wait until the job is complete before being able to move forward with the workflow. Since the comprehend job is asynchronous, **I created a wait time poller to periodically check the jobs status** using describe_targeted_sentiment_detection_job. Following a similar integration pattern as this https://docs.aws.amazon.com/step-functions/latest/dg/sample-project-job-poller.html. 3. However, there is seems to be a **Concurrent active asynchronous jobs quota of 10 jobs** according to https://docs.aws.amazon.com/comprehend/latest/dg/guidelines-and-limits.html#limits-active-jobs. If this is the case, I was thinking of **creating another poll to check if comprehend is free to do targeted sentiment before starting another comprehend job** 4. Given that the step functions charge for each polling cycle. And that there is a concurrent job limit of 10. I am worried about the backlog and respective costs that may be created if many step-function executions were to be started. For example, if 1000 workflows are started. Workflow number 1000 will have to be polling for an available comprehend job for a long time. Does anyone know if a solution is available to get around the concurrent active asynchronous jobs quota or to reduce the cost of step functions continually polling for a long time?lg...