Questions tagged with AWS Step Functions

Are nested step functions supported on AWS Step Functions Local? I am trying to create 2 step functions, where the outer one executes the inner one. However, when trying to execute the outer step function, getting an error: "The security token included in the request is invalid". To reproduce, use the latest `amazon/aws-stepfunctions-local:1.10.1` Docker image. Launch the container with the following command: ```sh docker run -p 8083:8083 -e AWS_DEFAULT_REGION=us-east-1 -e AWS_ACCESS_KEY_ID=TESTID -e AWS_SECRET_ACCESS_KEY=TESTKEY amazon/aws-stepfunctions-local ``` Then create a simple HelloWorld _inner_ step function in the Step Functions Local container: ```sh aws stepfunctions --endpoint-url http://localhost:8083 create-state-machine --definition "{\ \"Comment\": \"A Hello World example of the Amazon States Language using a Pass state\",\ \"StartAt\": \"HelloWorld\",\ \"States\": {\ \"HelloWorld\": {\ \"Type\": \"Pass\",\ \"End\": true\ }\ }}" --name "HelloWorld" --role-arn "arn:aws:iam::012345678901:role/DummyRole" ``` Then add a simple _outer_ step function that executes the HelloWorld one: ```sh aws stepfunctions --endpoint-url http://localhost:8083 create-state-machine --definition "{\ \"Comment\": \"OuterTestComment\",\ \"StartAt\": \"InnerInvoke\",\ \"States\": {\ \"InnerInvoke\": {\ \"Type\": \"Task\",\ \"Resource\": \"arn:aws:states:::states:startExecution\",\ \"Parameters\": {\ \"StateMachineArn\": \"arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorld\"\ },\ \"End\": true\ }\ }}" --name "HelloWorldOuter" --role-arn "arn:aws:iam::012345678901:role/DummyRole" ``` Finally, start execution of the outer Step Function: ```sh aws stepfunctions --endpoint-url http://localhost:8083 start-execution --state-machine-arn arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorldOuter ``` The execution fails with the _The security token included in the request is invalid_ error in the logs: ``` arn:aws:states:us-east-1:123456789012:execution:HelloWorldOuter:b9627a1f-55ed-41a6-9702-43ffe1cacc2c : {"Type":"TaskSubmitFailed","PreviousEventId":4,"TaskSubmitFailedEventDetails":{"ResourceType":"states","Resource":"startExecution","Error":"StepFunctions.AWSStepFunctionsException","Cause":"The security token included in the request is invalid. (Service: AWSStepFunctions; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: ad8a51c0-b8bf-42a0-a78d-a24fea0b7823; Proxy: null)"}} ``` Am I doing something wrong? Is any additional configuration necessary?

When using Step Functions with a container service like EKS or ECS how does output work for those steps? From the below two linked questions it seems as if input is done by either passing the input to the command override or by passing it as an environment variable. How do these jobs return output to be consumed by the following step? Input questions: * https://repost.aws/questions/QUQqIMl4s6QFKWtGeABPhgVA/step-functions-pass-input-into-fargate-instance * https://repost.aws/questions/QUCWMj_HPIQi6nmpUrbS3UTQ/step-functions-lambda-and-ecs **EDIT** According to this blog there isn't a way to output from ECS or Fargate tasks. So the outputs need to be deterministic based on the previous inputs. Can anyone confirm this and does this go for EKS as well? https://nuvalence.io/blog/aws-step-function-integration-with-ecs-or-fargate-tasks-data-in-and-out

I have the following state machine step: ``` "Step Name": { "Type": "Task", "Resource": "arn:aws:states:::sns:publish", "Parameters": { "TopicArn": "${TopicARN}", "Message.$": "States.JsonToString($)", "MessageAttributes": { "type": { "DataType": "String", "StringValue": "$.type" }, "id": { "DataType": "String", "StringValue": "$.id" } } }, "End": true } ``` which works fine but I'd like to customise the body of the SNS message, somehow like this: ``` "Step Name": { "Type": "Task", "Resource": "arn:aws:states:::sns:publish", "Parameters": { "TopicArn": "${TopicARN}", "Message.$": { "id": "$.id", "param1": "$.param1", "param2": "$.param2", "metadata": "$" }, "MessageAttributes": { "type": { "DataType": "String", "StringValue": "$.type" }, "id": { "DataType": "String", "StringValue": "$.id" } } }, "End": true } ``` However it does not work because `Message` has to be a `String`. I was hoping to wrap my object in `States.JsonToString` but that also doesn't seem to work. Is there a way to achieve this or will I have to resort to using a lambda instead of the SNS integration?

Is there a serverless, codeless way I can send an email with an attachment? Is there a way I can do this with Step Functions? The SES v2 API SendEmail action requires building the entire properly formatted and encoded MIME message. Is there any to build such a MIME message codelessly and serverlessly? Is my only option to drop the codeless requirement and use Lambda?

Is there an AWS service that can codelessly, serverlessly download an external file via HTTPS to an S3 bucket? Is there a way I can serverlessly, codelessly download a file from a given external HTTPS URL and save the downloaded file to a given S3 bucket? Is there a way I can do this with Step Functions? Is my only option to drop the codeless requirement and use Lambda?

I am a newbie so pardon my ignorance. I am writing a very simple step function state machine that uses the AWS SDK to retrieve a file from S3. Every time I run it the task that gets the file from S3 fails with an "S3.InvalidContent" error with "Failed to convert 'Body' to string" as the cause. The full definition of my state machine is: ``` { "Comment": "A description of my state machine", "StartAt": "GetAudioFile", "States": { "GetAudioFile": { "Type": "Task", "Parameters": { "Bucket": "11123", "Key": "test.wav" }, "Resource": "arn:aws:states:::aws-sdk:s3:getObject", "End": true } } } ``` The full text of the TaskFailed event is: ``` { "resourceType": "aws-sdk:s3", "resource": "getObject", "error": "S3.InvalidContent", "cause": "Failed to convert 'Body' to string" } ``` The full text of the CloudWatch log entry with the error is: ``` { "id": "5", "type": "TaskFailed", "details": { "cause": "Failed to convert 'Body' to string", "error": "S3.InvalidContent", "resource": "getObject", "resourceType": "aws-sdk:s3" }, "previous_event_id": "4", "event_timestamp": "1651894187569", "execution_arn": "arn:aws:states:us-east-1:601423303632:execution:test:44ae6102-b544-3cfa-e186-181cdf331493" } ``` 1. What am I doing wrong? 2. How do I fix it? 3. What additional information do you need from me? 4. Most importantly, where can I find answers to these stupid questions so I don't have to post these stupid questions on re:Post again? (I have spent nearly a day scouring AWS docs and Googling without finding anything.)

I've observed that the value of the AWS Step Functions context variable $$.State.EnteredTime will trim trailing zeros when the time is "on a second" (i.e. there are no milliseconds) rather than using ".000". This is okay according to ISO 8601, so within the realm of correctness for the Step Functions team. However, the DynamoDB .Net SDK is unable to deserialize the date in that format. So, when using a DDB SDK integration in a StateMachine that writes out the value of $$.State.EnteredTime (unaltered) to a table, attempting to deserialize it using a DynamoDBContext, will result in an exception being thrown. Is this a bug, and if so in which product? IMHO, the trimming of trailing zeros is a weird and essentially pointless optimization (if that's the motivation), but the brittleness of the DDB .Net SDK is likewise an issue. More importantly, is there a better way to handle this other than writing a custom IPropertyConverter and using it on any property that may be updated from a StateMachine?

We use Step Functions pretty extensively in one of our applications. I noticed higher than expected costs in CloudTrail and GuardDuty which caused me to investigate. It looks like every call to StartExecution, SendTaskHeartbeat, StartExecution, SendTaskSuccess, SendTaskFailure, etc are all considered "Management Events" inside CloudTrail. Since all of these function are normal usage of the Step Functions service, I think they should be considered "Data Events" in the same way that regular "usage" of S3, Dynamo, and Lambda API calls are handled. By being considered "management events", they are causing a large number of events (and cost) in CloudTrail, and similar with GuardDuty. Below is a typical event caused by an API call to SendTaskHeartbeat, where you can see `"managementEvent": true` and ` "eventCategory": "Management"`. I believe this should be `"managementEvent": false` and ` "eventCategory": "Data"` ``` { "eventVersion": "1.08", "userIdentity": { "type": "AssumedRole", "principalId": "xxxxxxxxxxxxxx-04fe38ef50d84dad1", "arn": "arn:aws:sts::722537357562:assumed-role/my-role-name/i-x0x4xfxex3x8xex", "accountId": "999999999999", "accessKeyId": "ASIAXXXXXXXXXXXPB", "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "AROxxxxxxxxxxxxAGI", "arn": "arn:aws:iam::999999999999:role/my-role-name", "accountId": "999999999999", "userName": "my-role-name" }, "webIdFederationData": {}, "attributes": { "creationDate": "2022-05-03T19:21:10Z", "mfaAuthenticated": "false" }, "ec2RoleDelivery": "2.0" } }, "eventTime": "2022-05-03T20:56:18Z", "eventSource": "states.amazonaws.com", "eventName": "SendTaskHeartbeat", "awsRegion": "us-east-1", "sourceIPAddress": "3.81.182.218", "userAgent": "aws-sdk-php/3.183.13 OS/Linux/5.4.0-1030-aws GuzzleHttp/6.5.5 curl/7.68.0 PHP/7.4.3", "requestParameters": { "taskToken": "AAAAKgAAAA......AqHoA+2qxXBI=" }, "responseElements": null, "requestID": "999999999-81de-40bf-8b77-7ccbf0db5fb4", "eventID": "999999999-2193-47dd-8e3d-10a5d9e6266d", "readOnly": false, "eventType": "AwsApiCall", "managementEvent": true, "recipientAccountId": "999999999999", "eventCategory": "Management", "tlsDetails": { "tlsVersion": "TLSv1.2", "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256", "clientProvidedHostHeader": "states.us-east-1.amazonaws.com" } }

Can one directly invoke a step function directly from a s3 object notification ? for example , when object is created. also, does it work similar to lambda , when hooked up to the s3 object notification. for each object created , will it start a new instance of step function or can we make it such that all of the object notification goes to the same instance of step function, and as i understand step function are state machines , so say i have two states enabled and disabled. when object is created, state machine/step function will move to enable state , and for any other s3 object created notification that comes after the first one , as the step function is already in a enabled state , it won't do anything unless it is in a disabled state. once the files are processed it will move to disabled state.

According to https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html#aws-glue-api-jobs-job-GetJob the Action GetJob (AWS Glue) should return CodeGenConfigurationNodes for a given JobName as input however it only returns the Job object. [EDIT] Now tried using the python function inside a Lambda: ``` #!/usr/bin/python # -*- coding: utf-8 -*- import json import boto3 glue = boto3.client(service_name='glue', region_name='eu-west-2', endpoint_url='https://glue.eu-west-2.amazonaws.com') def lambda_handler(event, context): job_name = event.get('JobName') job_details = glue.get_job(JobName = job_name) return json.dumps(job_details, indent=4, sort_keys=True, default=str) ``` Doesn't return CodeGenConfigurationNodes eithier, so it looks like it maybe a bug in the python function that the the step function is using?

``` { "Comment": "A description of my state machine", "StartAt": "Batch SubmitJob", "States": { "Batch SubmitJob": { "Type": "Task", "Resource": "arn:aws:states:::batch:submitJob.sync", "Parameters": { "JobDefinition": "arn:aws:batch:us-east-1:XXXXXXXXXXX:job-definition/clientcopyjobdef:1", "JobQueue": "arn:aws:batch:us-east-1:XXXXXXXXXX:job-queue/copyclientjq", "ContainerOverrides": { "Command.$": [ "dotnet", "CopyClientJob.dll", "$.input" ] }, "JobName.$": "$.input" }, "End": true } } } ``` I was trying to create this state machine, it was working fine if I am passing command to JobDefinition directly, but here I am truing to override the command and want to pass command parameter from state input, so trying to pass like above code. For `"JobName.$": "$.input"` it is working but for ``` "Command.$": [ "dotnet", "CopyClientJob.dll", "$.input" ] ``` It is not working, the command is passing as it is to aws batch without trasforming the parameter, can someone help on this. Thanks

I'm trying to start a backup job from a step function, but getting a Backup.BackupException, with this message: > Insufficient privileges to perform this action. (Service: Backup, Status Code: 403, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) My state is very simple ($.table is the ARN of a dynamoDB table): "StartBackupJob": { "Type": "Task", "Parameters": { "BackupVaultName": "my-vault", "IamRoleArn": "arn:aws:iam::xxxxxxxxxxxx:role/aws-service-role/backup.amazonaws.com/AWSServiceRoleForBackup", "ResourceArn": "$.table" }, "Resource": "arn:aws:states:::aws-sdk:backup:startBackupJob", "End": true } I even get this when the IAM role for the step function has full permissions. Also, if I assume this role, and use it to start a backup job from the AWS CLI with the same exact parameters, it succeeds. Any idea what I'm doing wrong?

I'm using SAM and I want to connect an HTTP POST endpoint from API Gateway to an AWS Step Function. I'm able to forward the request body but I'm unable to also pass the headers. This is my YAML definition: ``` /cancelOrder: post: responses: default: description: "zzzz" x-amazon-apigateway-integration: integrationSubtype: "StepFunctions-StartExecution" credentials: Fn::GetAtt: [zzzz, Arn] requestParameters: Input: "$request.body" StateMachineArn: Fn::GetAtt: [zzzz, Arn] payloadFormatVersion: "1.0" type: "aws_proxy" connectionType: "INTERNET" timeoutInMillis: 10000 ``` This works but I'm only able to receive the HTTP POST body, not the headers. Input: "$request.body" is, I suppose, the part that's parsing the JSON and sending it to my function. But I need both the Body and the HTTP headers. Is there a way to do this?

I want to be able to have a step function that does the following: 1. The step function starts with a call to API Gateway from my web app 2. The step function starts a Fargate pod based on a specific image 3. The step function runs a task on the pod it just created 3. After the Fargate task completes, terminate the pod so it does not consume resources Is this possible to tell Step Functions to start a new pod based on a certain image and then terminate that pod after it returns a success code? I have not been able to find any info on this. Thank you

How can we increase the internet speed of the container running in AWS ECS and later how can we verify that container internet speed is increased or not while downloading any dependencies on container or posting some data into S3 while working with JAR execution from the container etc. ?

I have a state that updates DynamoDB by shifting the first word off a list of words. It returns UPDATED_OLD so that the word can be grabbed in the ResultSelector block as index 0 and passed on to the next step. If there are words in the list this works fine, but if the list is empty, there is nothing at index 0 and Step Functions throws a States.Runtime error and the execution fails. I added a Catch block but it isn't catching as expected. Can this error be caught? Thank you.

Hi, I followed the wizard to create an ECS/fargate cluster and a basic step function state machine. I was able to run the state machine once (after working through a few permissions issues), though the container exited. I updated the task definition (specifically, all I changed was the container's entrypoint and command), and I'm now encountering a new IAM issue despite not (to my knowledge) changing anything related to the state machine or cluster's roles. ``` Error ECS.AccessDeniedException Cause User: arn:aws:sts::****:assumed-role/StepFunctions-hello-role-****/**** is not authorized to perform: ecs:RunTask on resource: arn:aws:ecs:us-west-2:****:task-definition/hello-task:2 because no identity-based policy allows the ecs:RunTask action (Service: AmazonECS; Status Code: 400; Error Code: AccessDeniedException; Proxy: null) ``` Is there a particular resource that needs to have this role/policy assigned that I'm missing? I don't know how to set or access permissions for "assumed roles" before or after the state machine runs. Thanks!

The [Saga Pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-the-serverless-saga-pattern-by-using-aws-step-functions.html) is a preferred way to do distributed transaction with Lambdas. The orchestration will work with lambdas. Please review the picture in the above link as well as this [image](https://ibb.co/56NL9Xy). My question is about if I use a **Cron Job** instead of one of the Lambdas to process **ProcessPayment**. Please see this [image](https://ibb.co/56NL9Xy). This **Cron Job** will retrieve the customers who have a flight and rental car and charge them *every 5 hours instead of immediately*. This 5 hours delay in the steps would block the whole function to be delayed for more than 5 hours. If that happens for a few hundreds customers, we will have* hundreds of transactions waiting to be committed or rolled back*. What is the right way to add *distributed transactional ability with Saga Pattern & Step functions* if we use functions that *don't run in sequential order and immediately*?

I have implemented LakeFormation on my data bucket. I have a step function in which one step consists of running a GlueJob that reads and writes to the data catalog. I have upgraded my DataLake permissions as reported [here][1]. The Service Role that runs my Step Function has a root-type policy (granted just for debugging this issue): ```yaml Statement: - Effect: "Allow" Action: - "*" Resource: - "*" ``` On lake formation the service role has: - Administrator Rights - Database Creation rights (and Grantable) - Data Location access to the entire bucket (and Grantable) - Super rights on read and write Database (and Grantable) - Super rights on ALL tables within above Databases (and Grantable). The bucket is not encrypted. But, somehow, its access to the tables is denied with the error: ``` (AccessDeniedException) when calling the GetTable operation: Insufficient Lake Formation permission(s) on table ``` What's really strange is that the Glue Job succeeds when writing to some tables, and fails on others. And there is no real substantial difference across tables: all of them are under the same S3 prefix, parquet files, partitioned on the same key. Given the abundance of permissions granted, I am really clueless about what is causing the error. Please, send help. [1]: https://docs.aws.amazon.com/lake-formation/latest/dg/upgrade-glue-lake-formation.html

It looks like the States Language specification contains fuzziness and is not up to date with the AWS Step Functions implementation. **Example:** Intrinsic Function 'States.Format' (https://states-language.net/): "The Value of the first MUST be a string". In your example on the following page (https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html), the first parameter can also be a JSONPath. Could you please update the State Language page (https://states-language.net/)? Thanks a lot!

Hello everyone, I am working on a poc to measure response time of Step function for various number of concurrent requests. Two approaches i have taken : (1) Execute Step function from API gateway (2) Execute step function from aws sdk java v2 I am using apache benchmark tool to trigger 500, 1000, 3000 and 5000 requests with 200 concurrent requests for each. The response time for api gateway scenario I am get is around 200 to 300 ms but for aws sdk I am getting response time in the range of 200 to 400 ms. I was expecting it to be other way around as sdk call should not be going through as api gateway and directly hitting the Step function StartSyncExecution api but strangely for sdk latency is more. For single request , aws sdk (avg res = 90 ms ) does perform better that apigaeway approach (avg res = 150 ms) I have created a rest service using spring boot and aws sdk to invoke the step function and testing this in one of the EC2 machine.I have used express work flow of Step function and it is orchestrating two api calls through lambda. No other logic is present inside Step function. Tried changing the tomcat max thread pool to 500 and used ApacheHttpClient to change max Http connection to 300 (from default of 50) but still it does not help. It actually worsens the latency and makes it > 1 sec. Any suggestions would be appreciated for optimization

Running a step function state machine accessing an AWS HttpApi to run a lambda using a docker container of ~3 GB size. Fails 20% of the time with #503 error, the rest of the time I get the exact results I expect. It's not a container warm up I believe. Logs for API and lambda (non-container and container) don't show any problems. I have maximized the step functions time out, the retries for the specific lambda, the memory for the lambda (10GB). I have also tried to ping the container and wait 30 seconds in the step functions to no avail. *** Edit: Should have mentioned concurrency for the API is at max (believe it is automatic for HttpApis) Any advice would be appreciated. I don't want to use ECS fargate due to cost and relative complexity. Thanks!

AWS Step function's Jsonpath implementation has some known limitations compared to the [original jsonpath implementation](https://github.com/json-path/JsonPath): > [Reference Paths](https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-paths.html) > A reference path is a path whose syntax is limited in such a way that it can identify only a single node in a JSON structure: > You can access object fields using only dot (.) and square bracket ([ ]) notation. > **The operators @ .. , : ? * aren't supported**. > **Functions such as length() aren't supported**. I'd like request to implement these unsupported features. Sample use-case: `Cognito User Pools: AdminGetUser` step-function action returns `UserAttrubutes` in an array of json nodes: ``` "UserAttributes": [ { "Name": "sub", "Value": "f3cab39d-bfe5-4b1a-a8fc-e1a97ec0d18b" }, { "Name": "email_verified", "Value": "true" }, { "Name": "custom:name", "Value": "My Name" } { "Name": "email", "Value": "myname@mycompany.com" } ] ``` Jsonpath filters would provide a very simple way of extracting data from this payload based on "Name" field of the nodes. For example getting the email for a user: `$.UserAttributes[?(@.Name=='email')].Value` Without this feature I needed to implement a lambda for this purpose, because I can't rely on the order of the response array.

I have a StepFunction with input: ```json { "Jobs": [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0] } ``` and I want the sum of the values in this array. According to the [Json-Path docs](https://github.com/json-path/JsonPath) there should exist a `.sum()` function for this. When I try it [here](http://jsonpath.herokuapp.com/) it even works. So I defined the following Pass state: ```json "Sum Jobs": { "Type": "Pass", "Parameters": { "Jobs.$": "$.Jobs.sum()" } }, ``` Nevertheless executions fail with: ``` "An error occurred while executing the state 'Sum Jobs' (entered at the event id #249). The JSONPath '$.Jobs.sum()' specified for the field 'Jobs.$' could not be found in the input '{\"Jobs\":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0]}'" ```

I have a state-machine workflow with 3 following states: [screenshot-of-my-workflow](https://i.stack.imgur.com/4xJTE.png) 1. A 'Pass' block that adds a list of strings(SageMaker endpoint names) to the original input. (*this 'Pass' will be replaced by a call to DynamoDB to fetch list in future.*) 2. Use map to call SageMaker endpoints dictated by the array(or list) from above result. 3. Send the result of above 'Map' to a Lambda function and exit the workflow. Here's the entire workflow in .asl.json, inspired from [this aws blog](https://docs.aws.amazon.com/step-functions/latest/dg/sample-map-state.html). ``` { "Comment": "A description of my state machine", "StartAt": "Pass", "States": { "Pass": { "Type": "Pass", "Next": "InvokeEndpoints", "Result": { "Endpoints": [ "sagemaker-endpoint-1", "sagemaker-endpoint-2", "sagemaker-endpoint-3" ] }, "ResultPath": "$.EndpointList" }, "InvokeEndpoints": { "Type": "Map", "Next": "Post-Processor Lambda", "Iterator": { "StartAt": "InvokeEndpoint", "States": { "InvokeEndpoint": { "Type": "Task", "End": true, "Parameters": { "Body": "$.InvocationBody", "EndpointName": "$.EndpointName" }, "Resource": "arn:aws:states:::aws-sdk:sagemakerruntime:invokeEndpoint", "ResultPath": "$.InvocationResult" } } }, "ItemsPath": "$.EndpointList.Endpoints", "MaxConcurrency": 300, "Parameters": { "InvocationBody.$": "$.body.InputData", "EndpointName.$": "$$.Map.Item.Value" }, "ResultPath": "$.InvocationResults" }, "Post-Processor Lambda": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "Payload.$": "$", "FunctionName": "arn:aws:lambda:<my-region>:<my-account-id>:function:<my-lambda-function-name>:$LATEST" }, "Retry": [ { "ErrorEquals": [ "Lambda.ServiceException", "Lambda.AWSLambdaException", "Lambda.SdkClientException" ], "IntervalSeconds": 2, "MaxAttempts": 6, "BackoffRate": 2 } ], "End": true } } } ``` As can be seen in the workflow, I am iterating over the list from the previous 'Pass' block and mapping those to iterate inside 'Map' block and trying to access the Parameters of 'Map' block inside each iteration. Iteration works fine with number of iterators, but I can't access the Parameters inside the iteration. I get this error: ``` { "resourceType": "aws-sdk:sagemakerruntime", "resource": "invokeEndpoint", "error": "SageMakerRuntime.ValidationErrorException", "cause": "1 validation error detected: Value '$.EndpointName' at 'endpointName' failed to satisfy constraint: Member must satisfy regular expression pattern: ^[a-zA-Z0-9](-*[a-zA-Z0-9])* (Service: SageMakerRuntime, Status Code: 400, Request ID: ed5cad0c-28d9-4913-853b-e5f9ac924444)" } ``` So, I presume the error is because "$.EndpointName" is not being filled with the relevant value. How do I avoid this. But, when I open the failed execution and check the InvokeEndpoint block from graph-inspector, input to that is what I expected and above JSON-Paths to fetch the parameters should work, but they don't. [screenshot-of-graph-inspector](https://i.stack.imgur.com/3gXsM.jpg) What's causing the error and How do I fix this?

A step function workflow I'm working on needs to call an external service and wait for it to complete before continuing, so I'm using WAIT_FOR_TASK_TOKEN integration. The external service does not do task token callbacks, it notifies via SNS. So we have a lambda subscribed to the SNS topic, and when it gets triggered it looks up the appropriate task token then calls SendTaskSuccess to continue the workflow. However, SNS does not guarantee that a notification will only be delivered once, it could show up multiple times. So what happens if I call SendTaskSuccess with a task token, then call it again with the same task token? My assumption is that it would throw an exception, but I have not found documentation stating that, or exactly what exception would be thrown. Any help would be appreciated

Here's some part of my StepFunction: https://i.stack.imgur.com/4Jxd9.png Here's the workflow for the "Parallel" node: ``` { "Type": "Parallel", "Branches": [ { "StartAt": "InvokeEndpoint01", "States": { "InvokeEndpoint01": { "Type": "Task", "End": true, "Parameters": { "Body": "$.Input", "EndpointName": "dummy-endpoint-name1" }, "Resource": "arn:aws:states:::aws-sdk:sagemakerruntime:invokeEndpoint" } } }, { "StartAt": "InvokeEndpoint02", "States": { "InvokeEndpoint02": { "Type": "Task", "End": true, "Parameters": { "Body": "$.Input", "EndpointName": "dummy-endpoint-name2" }, "Resource": "arn:aws:states:::aws-sdk:sagemakerruntime:invokeEndpoint" } } } ], "Next": "Lambda Invoke" }, ``` I would like to access the `EndpointName` of each node inside this Parallel block and add it as one of the keys of that particular node's output, without modifying the existing output's body and other headers.(in the above json, `EndpointName` can be found for first node inside the Parallel at `$.Branches[0].States.InvokeEndpoint01.Parameters.EndpointName`) Here's output of one of the node inside the Parallel block: ``` { "Body": "{xxxx}", "ContentType": "application/json", "InvokedProductionVariant": "xxxx" } ``` and I would like to access the API Parameter and make it something like below: ``` { "Body": "{xxxx}", "ContentType": "application/json", "InvokedProductionVariant": "xxxx", "EndpointName": "dummy-endpoint-name1" } ``` How do I do this?

Hi, We successfully trigger fargate task through step function, Something like below: * Is there anyway to define the ephemeral storage size, cpu and memory within the container override section below? * Now, when we create fargate task, we set desire_count=0, min_count=0 and we have 0 instance always on (which is good, we don't want to always on instance). But how will max_count be treated? When we have more than max_count of concurrent step function running, will some step just hanged and waiting for the finishment of certain fargate task? Or throwing error? Thank you , in advance. "UncompressGranuleECS": { "Parameters": { "LaunchType": "FARGATE", "Cluster": "${module.cluster.ecs_cluster_name}", "TaskDefinition": "${local.fargateTaskARN}", "NetworkConfiguration": { "AwsvpcConfiguration": { "Subnets": ${jsonencode(var.subnet_ids)}, "AssignPublicIp": "DISABLED" } }, "Overrides": { "ContainerOverrides": [ { "Name":"dyen-cumulus-uncompress_granule-fargate", "Command": ["java", "-Xms7500M", "-Xmx7500M", "-jar", "/home/dockeruser/build/libs/my-jarfile.jar"], "Environment":[ { "Name":"region", "Value":"us-west-2" }, { "Name":"TASK_TOKEN", "Value.$":"$$.Task.Token" }, { "Name":"CMA_MESSAGE", "Value":"myjson:[]" } ] } ] } }, "Type": "Task", "HeartbeatSeconds": 1800, "Resource": "arn:aws:states:::ecs:runTask.waitForTaskToken", "Retry": [ { "ErrorEquals": [ "States.ALL" ], "IntervalSeconds": 10, "MaxAttempts": 0 } ], "Catch": [ { "ErrorEquals": [ "States.ALL" ], "ResultPath": "$.exception", "Next": "failureStep" } ], "Next": "MissingStep" },

I'm trying to migrate from ModifyInstanceAttribute+StartInstances to RunInstances and I'm stumped for a reason why they use different formats for UserData. Figuring out creating blobAttributeValue was hard enough, but it worked as it does base64 encoding for you. The question is how do I base64-encode UserData for RunInstances in my state machine? What's more, even if I put base64-encoded data in UserData (callback task token), it never shows up in the instance's user data - in Web UI it is shown as "No user data found". Apparently just because the task token decodes into a random binary stream, and it causes problems down the line - I guess AWS tries to parse it to figure out if it's some script to execute and bails out completely. My main goal is to pass the task token to the EC2 instance I create (along with some other params which I can put in tags separately - it works). But the task token won't fit into a tag, so UserData seems to be the only option. I am at a loss - is it even technically possible to use RunInstances with a callback?

Hi Folks, I am working on a modernization project, which contains API and several executable jobs. Customer has configured these executable in windows scheduler and trigger these jobs via C# code inside the API in fire and forget manner. Some jobs are created in the form of workflow for which they have created seprate C# exe. also these jobs taking hours and days to execute. The tech stack is .Net 4.5 and SQL Server (Windows authentication). They want to use step functions to create workflow. The problem with legacy system is , it is executing requests one by one. The API sends the request which they are storing in DB and executables are picking these requests one by one and processing. We were planing to propose design using Step functions, Lambda and AWS batch for long running jobs but as lambda does not support .Net 45, it will require .Net core migration first. can you please suggest good design for this?

by looking at this AWS document: https://docs.aws.amazon.com/step-functions/latest/dg/sample-project-container-task-notification.html step functions can totally instantiate/manage fargate or ECS Task. Speaking about fargate * the example put fargate as first step. suppose fargate is the 2nd step, can we get the output message from first step (larger than 8K) into fargate task through either .sync or .waitForTaskToken model? Instead of passing input parameters through fargate containerOverride's command or commands? * If above is possible, does the code inside fargate suppose to do something like : GetActivityTaskResult getActivityTaskResult = client.getActivityTask(new GetActivityTaskRequest().withActivityArn(stringActivityArn)); String taskToken = getActivityTaskResult.getTaskToken(); String taskInput = getActivityTaskResult.getInput(); * In this step function instantiate fargate model (.sync or .waitForTaskToken), does fargate instances gets created each time when step functions "calls" it? That is, we actually would like to set desire_count and min count both to zero and forget about scaling alarms or metrics. If above is true, does it respect the max_count? I guess cost-wise, it is fine with us because how many concurrent fargate running does not matter anymore. Cost is based on usage. * is ECS or fargate able to return output values to the calling step function? that is, using sendSucess or some other api to send output back to step function for next step to use (become the input of next step)?

I have a lambda exposing a `GET /path/:parameter` endpoint and I want to invoke it as part of a Step Function state machine, taking the value of the `:parameter` from one of the input fields. Is this even possible? All I've been able to find in the documentation is how to specify JSON paths for input and output, and they always seems to be for `POST` calls.

Hi, Will I be charged for Fargate idle instance? We are going to run activity as Fargate task within a step functions. Scaling parameters min_count=1 desire_count=1 Max_count=17 Scaling up if average CPU usage more than 50% and scaling down if usage less than 50%. Questions are * The above setup has a min and desire count =1, will we be charged for the CPU/Memory/storage usage 24 hrs a day, 7 days a week? The activity program pulls message from a SQS queue and start processing and on most part of the day, it should be idle. * I understand, there is aws cli command to set desire_count=0 but the messages comes in randomly over the day (i.e., we can not turn desire_count >0 during a period of time and turn it off). Is there a better way to design the system to save cost? Thank you.

This question has come up before on the [AWS developer forums](https://forums.aws.amazon.com/thread.jspa?messageID=983086&tstart=0), but without any resolution. In our case, we have a step function that does the following: * Copy a Parquet file (generated by a separate process) from a temporary location in S3 to a permanent location via a lambda. This lambda blocks until the copy is complete. * Runs an Athena query on the copied file with the `arn:aws:states:::athena:startQueryExecution` state * Invokes a lambda that waits for the query to finish. This lambda checks the query status with the `get_query_execution` API call. We're receiving an error from the `get_query_execution` in ~4% of our step function executions: ``` HIVE_FILESYSTEM_ERROR: Incorrect fileSize 150486 for file s3://<redacted>.parquet, end of stream not reached ``` Like one of the commenters on the original issue, it seems to use like the copy is still in progress when the Athena query starts even though our lambda blocks until the copy had finished. We plan to workaround this issue for now by retrying on this error. But, I would still like to know: * Does anyone have any specific insight on why this happens? * Does AWS offer any guidance on how to prevent this issue? Thanks in advance!

I'd like to define a step that extracts secrets from secrets manager and then passes those secrets to another step. With logging enabled the secrets are logged as input to the next step. If I disable logging payloads, then other step's payload are also not logged. Is there a way to protect secrets between steps and still log other information?

Hi, I have a long running java program on fargate with 1024 unit cpu (1cpu) and 4G ram. serving as one of the step in step functions workflow. The java program has * An infinite loop pulling message from a SQS queue * download a 17G .tar.gz file using Transfermanager from S3 from another account (only 3+ minutes) * uncompress the .tar.gz file which will create a 50 G file and a small .md5 file with checksum value within. * verify the checksum of 50G file against the content from .md5 file. (about 7 minutes) * upload the untar-ed file to another S3 through multipart upload (upload multipart and then combine) which took around 12 mins. The problem is that, based on my log, I see the program just stopped without exception sometimes during un-tar, sometimes during upload. I am just guessing there could be a system level issue (OS) or JVM died. Can somebody please share your thought of * ways of debugging it? * Guess the problem(s) or any other suggestions? Thanks you.

Hi, I'm using step function with task DynamoDB:PutItem. When Step function try to save the JSON ``` { "TableName": "Table_Name", "Item": { "orderId": "159844346634674", "mail": "test@test.com", "itemId": "876434", "channel": "Retail", "createdDate": 1234512345 } ``` It receive the following error: could not be used to start the Task: [Cannot construct instance of `com.amazonaws.services.dynamodbv2.model.AttributeValue` (although at least one Creator exists): no int/Int-argument constructor/factory method to deserialize from Number value (1234512345)] The field createdDate into DynamoDB have type "N". Do you have any suggestion to fix this issue? Thanks

I created a simple test function with a task that invokes lambda. ``` Function weatherToday = Function.Builder.create(this, "WeatherToday") .code(Code.fromAsset(getClass().getClassLoader().getResource("lambda/weather_today").getPath())) .runtime(Runtime.NODEJS_14_X) .architecture(Architecture.ARM_64) .handler("WeatherToday.handler") .timeout(Duration.seconds(25)) .build(); IChainable getCurrentWeather = LambdaInvoke.Builder.create(this, "Get Current Weather") .lambdaFunction(weatherToday) .retryOnServiceExceptions(true) .build(); Pass hotState = new Pass(this, "It is HOT!"); Pass coldState = new Pass(this, "It is COLD!!"); Choice isItHot = new Choice(this, "Is it Hot?") .when(Condition.numberGreaterThanEquals("$.Payload.temp_f", 90), hotState) .otherwise(coldState); IChainable chain = Chain.start(getCurrentWeather).next(isItHot); StateMachine.Builder.create(this, "IsItHotNow").definition(chain).timeout(Duration.seconds(60))// best practice .build(); } ``` Assuming this is the best practice, I wanted to write a test that checks whether the states transition or are chained as intended. When pulling in the DefinitionString for the given state function ``` "DefinitionString": { "Fn::Join": [ "", [ "{\"StartAt\":\"Get Current Weather\",\"States\":{\"Get Current Weather\":{\"Next\":\"Is it Hot?\",\"Retry\":[{\"ErrorEquals\":[\"Lambda.ServiceException\",\"Lambda.AWSLambdaException\",\"Lambda.SdkClientException\"],\"IntervalSeconds\":2,\"MaxAttempts\":6,\"BackoffRate\":2}],\"Type\":\"Task\",\"Resource\":\"arn:", { "Ref": "AWS::Partition" }, ":states:::lambda:invoke\",\"Parameters\":{\"FunctionName\":\"", { "Fn::GetAtt": [ "WeatherToday53196B63", "Arn" ] }, "\",\"Payload.$\":\"$\"}},\"Is it Hot?\":{\"Type\":\"Choice\",\"Choices\":[{\"Variable\":\"$.Payload.temp_f\",\"NumericGreaterThanEquals\":90,\"Next\":\"It is HOT!\"}],\"Default\":\"It is COLD!!\"},\"It is COLD!!\":{\"Type\":\"Pass\",\"End\":true},\"It is HOT!\":{\"Type\":\"Pass\",\"End\":true}},\"TimeoutSeconds\":60}" ] ] } ``` The above is an array of strings with inlined intrinsic ASL functions. There does not seem to be a good way to get the state definitions to resemble the typical state function definition file as json without the intrinsic functions. My thoughts are if I can get the appropriate state machine definition as a json format of ASL it will be easier to test for the expected states. Please let me know if there is a way to do this or what may be the best practice approach?

Does Step Functions have metrics for how many times it retried to execute a state?

I have a state machine which needs to set EC2 instance's UserData. For that I'm using EC2 ModifyInstanceAttribute box and can't figure out how to set UserData (which has to be of type blobAttributeValue according to AWS docs). Example: ``` { "InstanceId.$": "$.Instance", "UserData": "Peppa Pig" } ``` Doesn't matter if I set it to "Peppa Pig" or its base64 representation, it fails with `Cannot construct instance of software.amazon.awssdk.services.ec2.model.BlobAttributeValue$BuilderImpl (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('Peppa Pig')` when I try to save the state machine

I'm working on SAML identification workflows in Step Functions where SAML messages has to be signed and returned Assertion is also encrypted. I will use KMS to store two different asymmetric keys (one for sign/verify and other for encrypt/decrypt) and tried to use for example 'kms:Sign' and 'kms:Decrypt' from SF through SDK integrations meaning task ARNs 'arn:aws:states:::aws-sdk:kms:sign' and 'arn:aws:states:::aws-sdk:kms:decrypt' but can only retrieve binary data in responses, which are not Base64-encoded. That's correct based on documentation: "When you use the HTTP API or the AWS CLI, the value is Base64-encoded. Otherwise, it is not Base64-encoded." Can I somehow always return Base64-encoded response or use binary response in context of SF json payloads? I can't figure out neither. Am I correct that SF can't decode/encode Base64? I also tried proxying through API gateway (which will use HTTP API as I think) but KMS responds always with 400 because CiphertextBlob can't be null. It isn't null, value is properly visible in step "request body payload after transformations" and I also can't figure out what prevents to call KMS through API gateway. If I will use Lambda to decode Base64 from request body, call KMS operation and encode Base64 from response body, all works nicely. Except including that SDK into Lambda code will increase total latency with multiple hundreds milliseconds because cold starts are much slower with SDK imported. Can I somehow avoid those overheads coming from Lambda and use KMS straight from SF or through API gateway?

Hi, I am trying to send a task-success request for my 'waitForTaskToken' step, but I see that the local step-machine server is waiting for the lambda to finish, instead of going to the next step how it works now on the AWS. At this moment task-success request from AWS CLI froze. **Local step-machine server does not return a response for 'send-task-success' request when it has been sent and waiting for the lambda to finish**. Step Functions Local Version: 1.8.1 (docker image) Step-function definition: ``` { "Comment": "", "StartAt": "waitForTaskToken", "States": { "waitForTaskToken": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken", "HeartbeatSeconds": 900, "TimeoutSeconds": 900, "Parameters": { "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:delay-function", "Payload": { "Token.$": "$$.Task.Token" } }, "End": true } } } ``` lambda 'delay-function' in this case - simple NodeJS function which prints the task token and waits 130 seconds. As we can see in the logs below: 1. 2022-02-09 10:16:25.078 - Executed local step-function 2. 2022-02-09T10:16:25.092Z - Received task token 'f68313c6-89bc-420f-b335-b8369dc5064e' and waiting 130000 ms 3. 2022-02-09 10:16:50.673 - I sent a task-success request from aws cli 'aws stepfunctions --endpoint-url http://localhost:8083 send-task-success --task-output '{"test": 123}' --task-token f68313c6-89bc-420f-b335-b8369dc5064e' after which this CLI command freezed, the **server does not return a response**. 4. 2022-02-09 10:16:50.673: - local step-function server received this request but does not return the result. [ISSUE THERE] does not stop the current step with the result from 'send-task-success' request until Lambda will not finishes. 5. 2022-02-09 10:18:35.100 - Lambda return data, (Duration: 130006.11 ms). 6. 2022-02-09 10:18:35.102 - Only now the local step-function server has returned the result for 'send-task-success' request. [ISSUE THERE] for some reason only now server returns a response for 'send-task-success' request instead of returning it at the same time it was sent (as it works now on AWS) Logs: ``` stepfunctions | 2022-02-09 10:16:25.078: [200] StartExecution <= {"sdkResponseMetadata":null,"sdkHttpMetadata":null,"executionArn":"arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273","startDate":1644401785077} stepfunctions | 2022-02-09 10:16:25.080: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"TaskScheduled","PreviousEventId":2,"TaskScheduledEventDetails":{"ResourceType":"lambda","Resource":"invoke.waitForTaskToken","Region":"us-east-1","Parameters":"{\"FunctionName\":\"arn:aws:lambda:us-east-1:123456789012:function:function\",\"Payload\":{\"Token\":\"f68313c6-89bc-420f-b335-b8369dc5064e\"}}","TimeoutInSeconds":900,"HeartbeatInSeconds":900}} stepfunctions | 2022-02-09 10:16:25.081: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"TaskStarted","PreviousEventId":3,"TaskStartedEventDetails":{"ResourceType":"lambda","Resource":"invoke.waitForTaskToken"}} lambda | START RequestId: b7e0b919-6519-4f84-bb36-70c540c8d550 Version: $LATEST lambda | 2022-02-09T10:16:25.092Z b7e0b919-6519-4f84-bb36-70c540c8d550 INFO >>> f68313c6-89bc-420f-b335-b8369dc5064e stepfunctions | 2022-02-09 10:16:50.673: SendTaskSuccess => {"requestClientOptions":{"readLimit":131073,"skipAppendUriPath":false},"requestMetricCollector":null,"customRequestHeaders":null,"customQueryParameters":null,"cloneSource":null,"sdkRequestTimeout":null,"sdkClientExecutionTimeout":null,"taskToken":"f68313c6-89bc-420f-b335-b8369dc5064e","output":"{\"test\": 123}","generalProgressListener":{"syncCallSafe":true},"readLimit":131073,"cloneRoot":null,"requestCredentials":null,"requestCredentialsProvider":null} stepfunctions | 2022-02-09 10:17:51.380: SendTaskSuccess => {"requestClientOptions":{"readLimit":131073,"skipAppendUriPath":false},"requestMetricCollector":null,"customRequestHeaders":null,"customQueryParameters":null,"cloneSource":null,"sdkRequestTimeout":null,"sdkClientExecutionTimeout":null,"taskToken":"f68313c6-89bc-420f-b335-b8369dc5064e","output":"{\"test\": 123}","generalProgressListener":{"syncCallSafe":true},"readLimit":131073,"cloneRoot":null,"requestCredentials":null,"requestCredentialsProvider":null} } "Token": "f68313c6-89bc-420f-b335-b8369dc5064e"7e0b919-6519-4f84-bb36-70c540c8d550 INFO EVENT: lambda | END RequestId: b7e0b919-6519-4f84-bb36-70c540c8d550 lambda | REPORT RequestId: b7e0b919-6519-4f84-bb36-70c540c8d550 Duration: 130006.11 ms Billed Duration: 130007 ms Memory Size: 3008 MB Max Memory Used: 3008 MB stepfunctions | 2022-02-09 10:18:35.100: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"TaskSubmitted","PreviousEventId":4,"TaskSubmittedEventDetails":{"ResourceType":"lambda","Resource":"invoke.waitForTaskToken","Output":"{\"StatusCode\":200,\"FunctionError\":null,\"LogResult\":null,\"Payload\":\"$LATEST\",\"ExecutedVersion\":null}"}} stepfunctions | 2022-02-09 10:18:35.101: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"TaskSucceeded","PreviousEventId":5,"TaskSucceededEventDetails":{"ResourceType":"lambda","Resource":"invoke.waitForTaskToken","Output":"{\"test\": 123}"}} stepfunctions | 2022-02-09 10:18:35.102: [200] SendTaskSuccess <= {"sdkResponseMetadata":null,"sdkHttpMetadata":null} stepfunctions | 2022-02-09 10:18:35.102: [200] SendTaskSuccess <= {"sdkResponseMetadata":null,"sdkHttpMetadata":null} stepfunctions | 2022-02-09 10:18:35.103: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"TaskStateExited","PreviousEventId":6,"StateExitedEventDetails":{"Name":"waitForTaskToken","Output":"{\"test\": 123}"}} stepfunctions | 2022-02-09 10:18:35.124: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"ExecutionSucceeded","PreviousEventId":7,"ExecutionSucceededEventDetails":{"Output":"{\"test\": 123}"}} ```

I opened up an AWS support case to increase my quota based on the error below, but AWS support states this is not a quota issue. They were unwilling to help me in support but directed me to this forum. I'm getting the following error when running some AWS SageMaker batch transform jobs on m5.large instances. I'm running 8 transforms in parallel against 8 different models in a step function map. A few of the transforms succeed, but some fail with the following error: { "resourceType": "sagemaker", "resource": "createTransformJob.sync", "error": "SageMaker.AmazonSageMakerException", "cause": "Rate exceeded (Service: AmazonSageMaker; Status Code: 400; Error Code: ThrottlingException; Request ID: 49a80dc1-df06-4b88-a462-24e517d13531; Proxy: null)" } What is going on here? Am I doing something wrong? What kind of throttling exception is this (what is being throttled?)

Using Step function AWS SDK integration for querying dynamodb table with key condition and filter condition, below is the query works fine outside of step-function. When trying to save the state machine after adding this, get the below Error. **Ddb Query** ``` { "TableName": "test", "ScanIndexForward": true, "KeyConditionExpression": "pk = :pk", "FilterExpression":"scheduled = :scheduled", "ExpressionAttributeValues": { ":pk": { "S": "test" }, ":scheduled": { "BOOL":true } } } ``` ** Error Saving State Machine ** ``` There are Amazon States Language errors in your state machine definition. Fix the errors to continue. The field "BOOL" is not supported by Step Functions For more information, see Amazon States Language ```

Hi, I am trying to create a step function in local using step-function-local docker images and ones the step function is created, I am using aws-cli docker image to create state machine, but I would like the aws-cli container to be alive so that I can do testing in my local environment, I tried setting tty option to true but that doesn't work. please help with achieving this, here is my docker-compose file. ``` version: "3.9" services: dynamo-db: image: "amazon/dynamodb-local" ports: - "8000:8000" step-function: image: "amazon/aws-stepfunctions-local" environment: - AWS_ACCESS_KEY_ID=<access-key> - AWS_SECRET_ACCESS_KEY=<secret-key> ports: - "8083:8083" state-machine: image: amazon/aws-cli:latest stdin_open: true # equivalent of -i tty: true # equivalent of -t network_mode: host #entrypoint: tail -F anything depends_on: - step-function deploy: mode: replicated replicas: 1 resources: limits: cpus: "2" memory: 2048M restart_policy: condition: on-failure delay: 5s max_attempts: 3 window: 120s env_file: - aws-stepfunctions-local-credentials.txt volumes: - ./config:/config command: stepfunctions --endpoint http://localhost:8083 create-state-machine --definition file:///config/<file-name> --name test --role-arn "arn:aws:iam::123456789012:role/HelloWorldFunctionRole" ```

1)I have 150 ETL jobs need to be moved from on-premise to AWS cloud. What is the best way to schedule them in AWS? AWS batch or SWF or STEP function or any other service? Which tool/service has all the capability of a full time job scheduler? 2) As per AWS Batch documentation, it says that it supports all jobs that are Docker container. Does it mean that AWS batch is only meant for Docker container jobs? In my case all jobs are either bash/ksh scripts and normal ETL jobs that read from files from EC2 and load data into RDS DB.

I am new to AWS. We have a yaml file defined like below to run our step function. I want to set a timeout for the step function in AWS parameter store and refer it in the file. I created a parameter from AWS console. But I am not sure how to refer it in the yaml file. After reading the documentation I understood that it is declared under "States" property as "Timeout". xxxStateMachine: Type: AWS::StepFunctions::StateMachine Properties: StateMachineName: !Sub ${AWS::StackName}-StateMachine RoleArn: !Sub arn:aws:iam::${AWS::AccountId}:role/roleName DefinitionString: !Sub - |- { "Comment": "Calls xxx state update job invoker lambda", "StartAt": "xxxCheckJob", "States": { "Timeout": ${}, // IS this where the timeout has to be defined? "xxxCheckJob": { "Type": "Task", "Resource": "${lambdaBDCArn}", "ResultPath": "$.isBusinessDay", "OutputPath": "$", "Next": "xxxCheckJobStatus" }, "businessDayCheckJobStatus": { "Type": "Choice", "Choices": [ { "Variable": "$.x", "BooleanEquals": true, "Next": "xxxStatexxJob" }, { "Not": { "Variable": "$.isBuxxy", "BooleanEquals": true }, "Next": "SuccessState" } ] }, "xxxStateUpdateJob": { "Type": "Task", "Resource": "${lambdaRSUArn}", "ResultPath": "$.detail.requestParameters.invocationStatus", "OutputPath": "$.detail.requestParameters", "Next": "xxxStateUpdateJobStatus" }, "xxxStateUpdateJobStatus": { "Type": "Task", "Resource": "${lambdaJSArn}", "Parameters": { "jobName.$": "$.invocationStatus.jobId" }, "ResultPath": "$.jobStatus", "OutputPath": "$", "Next": "checkJobStatus" }, "checkJobStatus": { "Type": "Choice", "Choices": [ { "Variable": "$.jobStatus.status", "StringEquals": "FAILED", "Next": "FailState" }, { "Variable": "$.jobStatus.status", "StringEquals": "SUCCEEDED", "Next": "SuccessState" }, { "Not": { "Variable": "$.jobStatus.status", "StringEquals": "SUCCEEDED" }, "Next": "Wait X Seconds" } ] }, "Wait X Seconds": { "Type": "Wait", "Seconds": 20, "Next": "xxxStateUpdateJobStatus" }, "SuccessState": { "Type": "Succeed" }, "FailState": { "Type": "Fail", "Cause": "Invalid response.", "Error": "ErrorA" } } } - {lambdaRSUArn: !GetAtt [ xx, Arn ], lambdaJSArn: !GetAtt [ xx, Arn ], lambdaBDCArn: !GetAtt [ xx, Arn ]} I have the following questions: 1. How to access timeout value in the yaml file. Is this the syntax - "Timeout": ${parameterName} 2. How to configure the step function so that it exits if the timeout reaches and job status is in pending state. 3. How to configure the step function so that it does not exits if the timeout reaches and the job status is in running state. Can anyone help me configure these?

Dear Sir/Madam, We have a couple of Step Functions running on a weekly basis, and we would like to know right after a Step Function encounter an error. Hence we place a "Send Email" Task (Amazon Pin Point Email Service) before the fail State.All the main task's error handling will Catch any error and continue the the "Send Email" State before failing the Step Function. in side the "Send Email" configuration, there is one API Parameters field, and i filled it in with : ``` { "Content": { "Body": { "Text": { "Data": "ERROR" } }, "Subject": { "Data": "ERROR" } }, "Source": "source@me.com", "Destination": { "ToAddresses": [ "notify@me.com" ] } } ``` However, when i try to save the state machine, i got an error saying: `The field "Body" is not supported by Step Functions` It seems like i dont know what need to be fill inside the API Parameters for Send Email task, unfortunately i cant find any reference for it. May i ask where to read up or refer for Step Function Send Email?

My state machine starts with a `MAP` state. Inside the `Map` state, there are 2 tasks, one after one (task#1 invokes lambda A, task#2 invokes lambda B). The lambda A will process the files conversion, upload those files into S3, and return a S3 object key. The lambda B will grab the S3 object key from task#1, get the object from S3, and then interact with the third party API. My question is that before the lambda B is invoked, is there a way to notify the state machine that the files have been saved in S3 and ready to be read? Could we implement the [S3-EventBridge](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html) in the asynchronous express workflows?

Having a real hard time getting data into Dynamo using Step Functions. The data flow is: 1. App Flow from Salesforce to S3 2. Event Bridge watching the S3 bucket 3. Map the data from S3 and store in Dynamo. This _almost_ works. Whenever App Flow gets new data from SalesForce, the step function is invoked. After the data is pulled from S3 (using GetItem), the next step is a Lambda in C#. This Lambda just deserializes the JSONL, populates a C# model for each item, and returns an array of the model. Next step is a Map step that passes each item in the Array to DynamoDB PutItem. Everything works, until you try to write a Boolean. Strings and Numbers are written fine, but it blows up if it sees a boolean. DynamoDB won't convert the boolean value. Things I have tried that didn't work: 1. Write the entire model object to Dynamo - blows up on bool values 2. Update PutItem config to map each field - blows up if there are any Nulls (nulls get dropped when the return value from the Lambda is serialized into JSON) 3. Serializing the values back into a JSON string inside the Lambda - blows up on bool values 4. Returning a list of DynamoDB Documents - blows up because each item in the document gets serialized as an empty object 5. Bypassing the Lambda altogether and try passing the JSONL to the Map step - blows up because it's not an array When trying write Bool values, the error is "Cannot construct instance of `com.amazonaws.services.dynamodbv2.model.AttributeValue` (although at least one Creator exists): no boolean/Boolean-argument constructor/factory method to deserialize from boolean value (false)" I can't see any obvious way to get this working, unless we convert _every_ item to a string, which causes problems reading the model back out later, as we lose type information and makes deserializing the model difficult.

I am sure I am missing some basic info but my setup is as follows: I have 1 lambda function and I have 2 State Machines. Each state machine is calling the same lambda function twice in sequence (30 sec pause between calls) Then I have rules setup to trigger the state machines every minute. Each state machine is passing different params to the lambda function so I am trying to get to a situation where my Lambda function is called every 30 seconds with 1 set of params (from statemachine 1), and the same lambda function is called with a different set of params (via statemachine 2) every 30 seconds. Looking at the lambda function logs it looks like the state machines will not run the lambda function until the other state machine has completed its entire execution (ie calling the lambda function twice). I would expect that the two state machines would run independently of each other. Is there some limitation because they are all calling the same lambda function? Or is there some setup issue or is this just how it works? Thanks!

I'd like to have the size of a Batch job be configurable from inputs of a Step Function. However, both: ``` "ArrayProperties": { "Size": "$.ArraySize" } ``` and ``` "ArrayProperties": { "Size": $.ArraySize } ``` do not work. The former complains that type, string, is invalid for the paramter, and the latter complains of a syntax error. Is it what I'd like to achieve possible?

HI All This is my first implementation of StateMachineFragment. Goal: Attempting to create a class for re-usable lambda state. This class can take a parameter and pass this as payload to Lambda and the lambda will execute the right query based on the payload. Below is my POC code to 'classs-ify' the lambda and the call to statemachine. ``` from aws_cdk import ( Duration, Stack, # aws_sqs as sqs, aws_stepfunctions as _stepfunctions, aws_stepfunctions as sfn, aws_stepfunctions_tasks as _stepfunctions_tasks, aws_lambda as _lambda, ) from constructs import Construct class SubMachine(_stepfunctions.StateMachineFragment): def __init__(self, parent, id, *, jobTypeParam): super().__init__(parent, id) existingFunc = _lambda.Function.from_function_arn(self, "ExistingLambdaFunc", function_arn="arn:aws:lambda:us-east-1:958$#$#$#$:function:dummyFunction") lambda_invoked = _stepfunctions_tasks.LambdaInvoke(self, "someID", lambda_function=existingFunc) wait_10_seconds = _stepfunctions.Wait(self, "Wait for 10 seconds", time=_stepfunctions.WaitTime.duration(Duration.seconds(10)) ) self._start_state = wait_10_seconds self._end_states = [lambda_invoked.end_states] def start_state(self): return self._start_state def end_states(self): return self._end_states class StepfunctionsClasStack(Stack): def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None: super().__init__(scope, construct_id, **kwargs) test_lambda_1 = SubMachine(self, "SubMachine1", jobTypeParam="one") state_machine = _stepfunctions.StateMachine(self, "TestStateMachine", definition=test_lambda_1, # role=marketo_role ) ``` When I try and deploy this code, I get the following error: ``` jsii.errors.JSIIError: Cannot read properties of undefined (reading 'bindToGraph') ``` I am not sure where I am going wrong. Thoughts? Thanks

Hi All All I am trying to do is to create a reusable ambda component where I can pass parameters to the class so that the Lambda can do different things, based on input param. I am using CDK in python to deploy the stack. I would like to create a parallel stepfunction, where I can pass the same lambda using different param/payload so they can be branches. I am running the following code: ``` from aws_cdk import ( # Duration, Stack, # aws_sqs as sqs, aws_stepfunctions as _stepfunctions, aws_stepfunctions_tasks as _stepfunctions_tasks, aws_lambda as _lambda, ) from constructs import Construct class LambdaJob(_stepfunctions.StateMachineFragment): def __init__(self, parent, id, *, jobTypeParam): super().__init__(parent, id) existingFunc = _lambda.Function.from_function_arn(self, "ExistingLambdaFunc", function_arn="arn:aws:lambda:us-east-1:95842$$$$$:function:dummyFunction") lambda_invoked = _stepfunctions_tasks.LambdaInvoke(self, "someID", lambda_function=existingFunc) wait_10_seconds = _stepfunctions.Wait(self, "Wait for 10 seconds", time=_stepfunctions.WaitTime.duration(Duration.seconds(10)) ) self.start_state = wait_10_seconds class StepfunctionsClasStack(Stack): def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None: super().__init__(scope, construct_id, **kwargs) test_lambda_1 = LambdaJob(self, "Quick") #state_machine = _stepfunctions.StateMachine(self, "TestStateMachine", # definition=LambdaJob(self, "Quick"), # # role=marketo_role # ) ``` However i keep getting the following error: ``` TypeError: Can't instantiate abstract class LambdaJob with abstract methods end_states, start_state ``` Any thoughts on what I am doing wrong ? Thanks

Hello All I am expanding the scope of the project, and wanted some suggestions/comments on if the right tech stack is being used. (Job - Pulling Leads and Activities from Marketo). Each job has a different query. **Scope**: We need 2 jobs to be run daily, however here is the catch. Each job should be created and queued first. Once done, we can poll to see if the job on Marketo side is completed and the file is downloaded. The file download can take more than 15 mins. THe file an be downloaded using the Job Id, using the 2 jobs that were created earlier. **Current/Implemented Solution**: I started with solving for Leads only and here is the stack that was worked out. The Event will be triggered, on a daily basis using event bridge. The task that is to be triggered is a step function. The Sfn, first calls a Lambda to create job, waits for 1 min, another lambda to queue the job, then wait for 10 mins and 3rd lambda to check status of file. If not ready, wait for 10 and poll again for file status(this is a loop with choice to keep checking file status). Once file is ready, call a container(fargate/ECS) and pass the Job Id as containerOverride to the container. Run the job on container to download the file and upload the file to S3. **Incorporate pulling activities into the above flow**: Since the queuing and Polling(for file status) lamba are similar, and the first lambda(creating the job) is different, I though of creating a parallel task where each branch does create, queue, poll and download the file(using the implemented solution, so 2 lambdas for job create and reuse the 2nd and 3rd lambdas). Once the complete chain(one for Leads and one for activities) is done, have a consolidation stage where the output from each container output is collected and an SNS message of job completion is send. I am looking forward to your suggestions to see if the suggested above workflow is how it should be done or is there any other technology that I should use. **Design Constraints**: I need to create and queue all the jobs first before starting the downloading since Marketo has a limit of 500mb for file download. Hence I the need to create and queue all the jobs first , and then only start the job to download of files. Thanks for your suggestions. Regards Tanmay

Hello, I'm having troubles inserting an item in DynamoDB with this Structure from Step Functions. ``` { "id":{ "S":"" }, "state_machine_execution_id":{ "S":"" }, "timestamp":{ "S":"" }, "map_param1":{ "M":{ "sub_map1":{ "M":{ } }, "sub_map2":{ "S":"" }, "sub_map3":{ "M":{ } }, "sub_mapN":{ "M":{ } }, "List_1":{ "L":[ ] }, "List_2":{ "L":[ ] } } } } ``` This is the statement that I am trying to make it work without success ``` { "DynamoDB Save Item":{ "Type":"Task", "Resource":"arn:aws:states:::dynamodb:putItem", "Parameters":{ "TableName.$":"$.TableName", "Item":{ "id.$":"$.id", "state_machine_execution_id.$":"$$.Execution.Id", "timestamp.$":"$.timestamp", "map_param1":{ "M":{ "M.$":"$.var_for_map_params", "List_1":{ "L.$":"$.List_1" }, "List_2":{ "L.$":"$.List_2" } } } } } } } ``` I'm getting this error ``` The Parameters "{A lot of data flow} "could not be used to start the Task: [The field \"foo\" is not supported by Step Functions]" ``` This "foo" field, has nothing to do, if make changes to the Parameters: map_param1 statement in Step Function works example This works ok ``` "map_param1":{ "M.$":"$.var_for_map_params" } ``` And This works ok too ``` "map_param1":{ "M":{ "List_1":{ "L.$":"$.List_1" }, "List_2":{ "L.$":"$.List_2" } } } ``` But not this (the combination) ``` "map_param1":{ "M":{ "M.$":"$.var_for_map_params", "List_1":{ "L.$":"$.List_1" }, "List_2":{ "L.$":"$.List_2" } } } ``` Can someone help me? thanks a lot Regards

Hello All I have a step function that is calling a lambda, which does some processing, and gets the id of the job that needs to be processed by the ecs runtask. What I am trying to do is to pass the job Id as containerOverride so that each run a different Id can be passed to the ecs Run task. Here is the dummy lambda Output: ``` Test Event Name dummyTest Response { "containerOverrides": " [ { \"name\": \"getFileTask\", \"environment\": [ { \"name\": \"name1\", \"value\": \"123\" }, { \"name\": \"DATE\", \"value\": \"1234-12-12\" }, { \"name\": \"SCRIPT\", \"value\": \"123456\" } ] } ] " } ``` Dummy Lambda Code: ``` def lambda_handler(event, context): # TODO implement print("Success") overridden_Text = ' [ { "name": "getFileTask", "environment": [ { "name": "name1", "value": "123" }, { "name": "DATE", "value": "1234-12-12" }, { "name": "SCRIPT", "value": "123456" } ] } ] ' return{ 'containerOverrides': overridden_Text } ``` Here is the record when the ECS run task is triggered(TaskStateEntered) from the step function: ``` { "name": "ECS RunTask", "input": { "containerOverrides": " [ { \"name\": \"getFileTask\", \"environment\": [ { \"name\": \"name1\", \"value\": \"123\" }, { \"name\": \"DATE\", \"value\": \"1234-12-12\" }, { \"name\": \"SCRIPT\", \"value\": \"123456\" } ] } ] " }, "inputDetails": { "truncated": false } } ``` The issue is that when the run task enters the TaskSubmitted stage: ``` "LastStatus": "PROVISIONING", "LaunchType": "FARGATE", "Memory": "4096", "Overrides": { "ContainerOverrides": [ { "Command": [], "Environment": [], "EnvironmentFiles": [], "Name": "getFileTask", "ResourceRequirements": [] } ], "InferenceAcceleratorOverrides": [] }, "PlatformFamily": "Linux", "PlatformVersion": "1.4.0", "StartedBy": "AWS Step Functions", ``` For what ever reasons the Environment variable is not being pushed from lambda output to the Container as launch override. Is there something that I am doing incorrectly ? Thanks

I'm working on a step function state machine and can create lambdas in python and node to update an existing item in ddb. However, I can't seem to find any examples with service integrations AND dynamic variables. The only ones I come across have static variable inserts/updates. In my case I am passing in data from an upstream lambda in python (which I am not so experienced with). Seems straightforward but I think my issue is json vs. dynamodb json. Thanks in advance.

I have an endpoint set up on API Gateway with integration to a step function - the integration is working well and my function is executed. However I have a need to access the headers on the initial request to the api gateway (as they need to be passed on to an API call made by one of the steps in the step function) I've added the http headers to the API Gateway Method Request and also done this in the HTTP Headers section of the Integration Request, then in the mapping template I have ``` #set($inputbody = $input.json('$')) { "method": "$context.httpMethod", "input": "$util.escapeJavaScript($inputbody)", "stateMachineArn": "MyStateMachineARN", "headers": { #foreach($param in $input.params().header.keySet()) "$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end #end } } ``` When I test this I see the headers in the logs after the request body has been transformed - before it executes the step function ``` request body after transformations: { "method": "POST", "input": "{\"surname\":\"TESTSURNAME\"}", "stateMachineArn": "MyStateMachineARN", "headers": { "HeaderA": "ValueA" , "HeaderB": "ValueB" } } ``` But in the step functions I am struggling to see the headers - the input I can see at the start of the execution is only ``` { "surname::"TESTSURNAME" } ``` I have inputPath set to $ and the same for the payload. All the suggestions I've found online point to the mapping template but I can not get it to work - any ideas what I'm doing wrong?

We are using Step Functions for our ETL pipeline. The first step kicks off 21 jobs that each take about 1-3 minutes each consuming 5 DPUs. The Step Function fails with the below error when trying to run more than 15 Glue Jobs in parallel. We are using the arn:aws:states:::glue:startJobRun.sync task to invoke the jobs synchronously. Is there a quota I need to ask for an increase on? Kicking off 21 jobs in parallel seems pretty reasonable. { "resourceType": "glue", "resource": "startJobRun.sync", "error": "Glue.AWSGlueException", "cause": "Rate exceeded (Service: AWSGlue; Status Code: 400; Error Code: ThrottlingException; Proxy: null)" }

Hello All Here is what I am attempting to do using StepFunctions/Lambda/ECS+Fargate+python+docker I have a job that is queuing a request and polling to see if the request is complete. Once complete, the file for the request needs to be downloaded. All the above steps, except downloading file is done using Lambda and Choices in Step functions. For the actual download I am using a container as the download can take over 15 mins. Question: What I need to do is send the Job_ID from The final "Is File Ready?" Choice to the Fargate/ECS Container so that the python script in the container downloads the file and puts it in S3. The final choice, is basically checking if the file status is Ready or Pending. If Ready, the file is ready to download, else keep looping after 10 mins sleep and check the status again. Is this the correct thought process: Somehow in the "Is File Ready?" state the output should include the json that the ECSTaskRun can consume? and this json output should have the Job_id as an evn parameter? Once this is done, the python script can read the job_id as env param, os.environ.get('job_id), and do the remaining items? Or, am I on the wrong track here. Also I am deploying everything in Python CDK. Thanks for the feedback.

In the [Exponential Backoff And Jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) aws architecture blog article it discusses the importance of having jitter as part of your retry functionality. I am currently using the baked in retry functionality with the "BackoffRate" parameter for "exponential backoff" that step functions provides. However, as far as I can tell jitter is not implemented out of the box as part of that option. Is there a way to get this functionality from the as-is step functions implementation?

What is the best way to collect EB events and Step Function State Machines status, preferably in one place and on a shoestring budget? My scenario: Working on an MVP whereby Eventbridge launches a bunch of Step Function State Machines. These grab data from DDB, do some text processing, post results back to the table then put the status of the completed job to SNS (the topic emails me although I might switch to Slack) and lastly back to EB. Of course everything is logged in Cloudwatch in (mostly) individual logs. Clearly this is a flood of notifications but as this is my first MVP with serverless, and soon to launch, I want to stay on top of things. Are there better solutions?

Hello, I am using AWS Step Functions with the new AWS SDK integrations. I am wondering how to deal with API endpoints which return only a limit number of items like for instance "rds:describePendingMaintenanceActions" so that following requests using the "NextToken" value are required to get all data. Is there any construct/suggestion available how to model this. I am aware that I can use a *Choice* state in order to check if "NextToken" is empty or not and to act accordingly (go back to the API request step and repeat the whole process with the *NextToken* value) but this feels somehow clumsy. Thank you very much for any information.

We currently run into the Problem, that some data analytic workloads exceed the 15 minutes timeout of a lambda. It is a multistep process with steps that are parallelizable and some that are not. The main driver for the runtime is a list of regular expressions that need to run sequentially over each of the provided data items, this step is parallelizable (by splitting the input data), but the next step needs to have all the input data. Input and output data is stored in a postresql RDS. Now to the Question: what is the right scalable approach to tackle this problem? Is it: having one 'main lambda' that spawns synced child-lambdas to speedup the parallelizable steps? But the compilation of the regular expressions takes some time, so the spawning of multiple child lambdas will most likely be not feasable. Or is it building a data pipeline using AWS Glue? Or with Steps? Or even building a data stream using Kinesis? I think using EMR is a little bit too big for that use case. Some metrics of the usecase: * size of the regular expression list ~10Mb in RAM * compilation time of all regular expressions around 5-10 sekunds * runtime of a single item going through all steps is about 100ms, but some requests need to process >30k items, therefore we run into the timeout. Any help in choosing the right architecture will be greatly appreciated Alexander

What are the key points to choose one of the following: * Data pipeline, * Step function * Amazon Managed Workflows for Apache Airflow

Hello, I was wondering how billing works for the 'map' state in step functions. Let's say I have an array with 100 items and they are processed by map state. States in map state are; 1- pass 2- wait 3- pass so in this case there are at least 300 transitions, right? Meaning almost 300 times more expensive than handling all items in a lambda function with a 'for loop' or something similar. I could not find any documentation about the billing of map state. I hope somebody can help me out with this. Thanks!

After "Announcement: AWS Step Functions adds support for over 200 AWS Services with AWS SDK Integration": <https://forums.aws.amazon.com/ann.jspa?annID=8933> We wanted to use the new AWS SDK Integration in Step Functions to start up a OpenSearch/Elasticsearch Domain, however Step Functions does not seem to support the **EBSOptions** parameter <https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configuration-api.html#configuration-api-datatypes-ebsoptions>, as it errors with: The Parameters '{\"ElasticsearchClusterConfig\":{\"InstanceType\":\"t3.small.elasticsearch\",\"InstanceCount\":1,\"DedicatedMasterEnabled\":false,\"ZoneAwarenessEnabled\":false},\"DomainName\":\"test-12345\",\"ElasticsearchVersion\":\"7.1\",\"EBSOptions\":{\"EBSEnabled\":true,\"VolumeType\":\"gp2\",\"VolumeSize\":100},\"VPCOptions\":OMITTED,\"AccessPolicies\":OMITTED}' could not be used to start the Task: The field \"EBSOptions\" is not supported by Step Functions The same parameters work just fine using boto3. Is this by design? Should we expect support in the future or it is never going to be an option? Is there any documentation about these limitations? Is there a workaround other than using the SDK directly in a Lambda? Edited by: LPKTech on Oct 22, 2021 5:15 AM - Fix formatting of the error and added detail about it running on boto3

Hi - I have a step function set up that invokes a glue job. This happens successfully, and the glue job succeeds in about 30 seconds. The step function state simply stays on `running`, never moving to the next state. I've waited about an hour after the glue job succeeded. Here is my state for the glue job: "Glue-Transform": { "Type": "Task", "Resource": "arn:aws:states:::glue:startJobRun.sync", "Parameters": { "JobName": "job-name", "Arguments": { "--argument.$": "$" } }, "Catch": \[{ "ErrorEquals": \["States.DataLimitExceeded", "States.Runtime", "States.Timeout", "States.TaskFailed", "States.Permissions"], "ResultPath": "$.Error", "Next": "MapFailed" }], "Catch": \[{ "ErrorEquals": \["States.ALL"], "ResultPath": "$.Error", "Next": "MapFailed" }], "End": true } I haven't been able to find documentation on troubleshooting anything related to this, besides this page which only details how to start a simple job, which I believe I am following correctly: https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html

I have tried examples from the docs, the ASL spec, and SO, nothing seems to give the expected result. It's either nothing or a literal string of the function call.

Can we use Step Functions to orchestrate a Sqoop job? The goal is to create a transient cluster which load data with Sqoop first followed by transformation in Hive, but looks like [Command Runner](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-commandrunner.html) does not have such option. If not, what are the alternatives?

Do Lambda functions called inside Step Functions still have a 250MB package limit?

I have a state machine splitted in some different lambdas to complete a process. One of the steps is a call to apigateway lambda function. Previous call is a call to lambda function and in ASL language I tell to pass the object overrided by this function to the next step. When it comes to Apigateway step, PathParameters are filled ok with the jsonpath accessors but when I try to use in RequestBody the received value are always empty. I'm using golang language State Machine "UploadFile": { "Type": "Task", "Resource": "${UploadFileFunctionArn}", "Retry": \[ { "ErrorEquals": \[ "States.TaskFailed" ], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0 } ], "Catch": \[ { "ErrorEquals": \[ "States.ALL" ], "ResultPath": null, "Next": "Fail" } ], "ResultPath": "$", "Next": "UpdateTerminal" }, "UpdateTerminal": { "Type": "Task", "Next": "HasBeenUpdated", "Resource": "${UpdateTerminalFunctionArn}", "Parameters": { "ApiEndpoint": "${APIDomainName}/${APIBasePath}", "Method": "POST", "Stage": "${ApiStageName}", "Path": "", "PathParameters": { "terminalId.$": "$.terminalId" }, "RequestBody.$": "$", "AuthType": "NO_AUTH" }, "Retry": \[ { "ErrorEquals": \[ "States.TaskFailed" ], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0 } ] }, Update terminal lambda logs : fmt.Println("request.Body", request.Body) fmt.Println("request.PathParameters\['terminalId']", request.PathParameters\["terminalId"]) 2020-12-07T13:55:23.889+01:00 request.Body 2020-12-07T13:55:23.889+01:00 request.PathParameters\['terminalId'] dc725a3a-0b96-4b14-8036-5c36514466556 Edited by: eguzkilorebeltza on Dec 7, 2020 5:08 AM

What is the difference between SageMaker Pipelines and SageMaker Step Function SDK? The [official Pipelines notebook][1] is basically only doing a workflow - pretty much a copy cat of what step functions has been doing for years. In the nice video from [Julien Simon][2] I see CICD capacities mentioned, where are those? any demos? [1]: https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-pipelines/sagemaker-pipelines-preprocess-train-evaluate-batch-transform.ipynb [2]: https://www.youtube.com/watch?v=Hvz2GGU3Z8g

Hi. I'm using nested Step Function executions using the Run Job (sync) integration. The default child Execution Name generated by AWS is a generic UUID. I'd like to specify my own (more meaningful) execution name, by passing it as an input attribute to the parent Step Function. I tried to use a "Name": "$.executionName" parameter in the integration task, but I got an Invalid Name error when starting the child Step Function execution. I passed an alphanumeric/underscore input value for "executionName", which should be legal as a Step Function execution name. I suspect Step Functions treats the "Name" parameter as a literal value and doesn't expand the reference, and tries to create an execution with the literal name "$.executionName". This raises the question, how else can I specify dynamic execution names for nested Step Functions? If the parameter is a literal string, it'll only work once for a single execution, which doesn't make much sense. PS. I'm using AWS CDK and the StepFunctionsStartExecution construct.

I'm using Step Functions to create transient Amazon EMR clusters. For the task node, I use a Spot instance, and for timeouts, I have set up a rule to terminate the cluster. Can I use multiple spot rules in the Step Function script for creating clusters with different specifications (example: i3.8xlarge and i4.8xlarge)? Using the Amazon EMR console, its possible to mention up to five instance types in the instance fleet. Is there a sample script that can help me test with multiple instance types? I'm referencing [Call Amazon EMR with Step Functions][1] to create my cluster. [1]: https://docs.aws.amazon.com/step-functions/latest/dg/connect-emr.html

I have a parameter 'request_id' that I want to have get an ID from two different formats of JSON that get sent to the input of my step function task. The first form looks like this: ``` { "request_id": "abcde-abcd-abcde-abc" } ``` The second is in this form: ``` { "request": { "id": "abcde-abcd-abcde-abc", } ``` Currently, I have a parameter which looks like ``` "request_id.$": "$.request_id" ``` but would like something that is equivalent to (this one does not work) ``` "request_id.$": "$.['request_id','request.id']" ``` The step function task looks like this ``` "Processing Error": { "Type": "Task", "Resource": "${UpdateRequestStatus.Arn}", "Parameters": { "status": "ERROR", "status_reason": "Failed to process data.", "domain_name.$": "$.domain_name", "client_id.$": "$.client_id", *"request_id.$": "$.request_id"* }, "Next": "Processing Failed" ``` Is this possible within the step function or will I need to either split those two request id's up into two paths in my JSON or do it in a function?

Customer has a question about using Lambda: *"Do you have any best practice/recommendations/online material for using Lambda and specifically how one can manage it through their landscape. At the moment, we manually turn the Lambda functions off before a release and then back on manually."* Is there a recommendation on how they can optimize this?

HI All, I have a step function to call an AWS Glue job with below step definition: ```json "glue-redshift-call": { "Type": "Task", "Resource": "arn:aws:states:::glue:startJobRun.sync", "Parameters": { "JobName": "GlueJobRedshiftStoredProcCa-meJ8P3QMSC5p", "Arguments": { "--SQLStatement": "call dw_merge_vendor('TNG', 'ASENTINEL')" } }, "Next": "post-processing" }, ``` As shown above, SQLStatement parameter is hardcoded as `call dw_merge_vendor('TNG', 'ASENTINEL')` in this. But I need to make it dynamic, parameter based. As mentioned in [Step Function new features Blogpost](https://aws.amazon.com/blogs/aws/aws-step-functions-adds-updates-to-choice-state-global-access-to-context-object-dynamic-timeouts-result-selection-and-intrinsic-functions-to-amazon-states-languages/), I tried below syntax to make it dynamic, but it's not working. Would you be able to suggest the correct syntax if I am doing something wrong here? ```json "Arguments": { "--SQLStatement": "States.Format('call dw_merge_vendor('{}', '{}'), ', $.param1, $.param2)" } ```

I'd like to notify an external observer (a client web app in my current case, but interested in other use cases too) of status updates when a Step Functions-orchestrated workflow progresses - in an event-driven way rather than by polling the execution history API.

I can see step function history wipes out after 90 days as per documentation. But due to nature of user information and analysis for production support, i want execution history to be retrieved older than 90 days. Is this provided by aws, or do we have to make provision out of the box to store it some where explicitly. and if we do so, will we be able to get pictorial representation of the execution. thanks in advance.

I have a (Python) Lambda function which returns a JSON object. I'd like to join that object on to my Step Functions state, and access its sub-fields in the next state... Something like: ```python # Lambda Function def handler(event, context): return json.dumps({ "ThingARN": "...", "AnotherField": "..." }) ``` ```json { "StartAt": "LambdaStuff", "States": { "LambdaStuff": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "MyCoolLambdaFunction", "Payload.$": "$" }, "ResultPath": "$.LambdaResult", "Next": "NextActivity" }, "NextActivity": { "Type": "Task", "Resource": "...", "Parameters": { "ARN.$": "$.LambdaResult.ThingARN" }, "ResultPath": "$.AnotherResult", "End": true } } } ``` ...But my state machine is failing (cancelling the `NextActivity` task) because what gets populated to `$.LambdaResult` is a big object with Lambda metadata and the ***stringified*** result payload - like this: ```json { "ExecutedVersion": "$LATEST", "Payload": "{\"ThingARN\": \"...\", \"AnotherField\": \"...\"}", "SdkHttpMetadata": { "HttpHeaders": { ... }, // All kinds'a stuff "HttpStatusCode": 200 }, "SdkResponseMetadata": { "RequestId": "..." }, "StatusCode": 200 } ``` [This docs page][1] seems to imply that what I'm trying to do is possible, but the example is clearly very simplified/stripped-down. All the other worked examples I've found (e.g. ["InputPath, OutputPath and ResultPath Example"][2]) just seem to have Lambdas returning simple strings, rather than structured objects. I guess I'm missing something obvious? [1]: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-state-machine-data.html#concepts-state-machine-data-state-input-output [2]: https://docs.amazonaws.cn/en_us/step-functions/latest/dg/input-output-example.html

I have created trivial step function to add a step to an EMR cluster. The jobflowid is not being recognised. Does anyone have a working solution here? Or pointer to what set up may be missing? The state machine: { "StartAt": "add_emr_step", "States": { "add_emr_step": { "End": true, "InputPath": "$", "Parameters": { "ClusterId": "j-INTERESTINGID", "Step": { "Name": "$$.Execution.Id", "ActionOnFailure": "CONTINUE", "HadoopJarStep": { "Jar": "command-runner.jar", "Args": [ "spark-submit --deploy-mode cluster s3://myfavouritebucketname86fda855-nhjwrxp55888/scripts/csv_to_parquet.py" ] } } }, "Type": "Task", "Resource": "arn:aws:states:::elasticmapreduce:addStep.sync", "ResultPath": "$.guid" } }, "TimeoutSeconds": 30 } Even with admin permissions the following error is returning. The EMR cluster is marked as visible. Copying and pasting the jobflowid into the console, it shows as expected. > Error > > EMR.AmazonElasticMapReduceException > > Cause > > Specified job flow ID does not exist. (Service: > AmazonElasticMapReduce; Status Code: 400; Error Code: > ValidationException; Request ID: be70a470-ef6d-4356-98cd-0fc36d2cd132) The IAM Role: { "Version": "2012-10-17", "Statement": [ { "Action": [ "elasticmapreduce:AddJobFlowSteps", "elasticmapreduce:DescribeStep", "elasticmapreduce:CancelSteps" ], "Resource": "arn:aws:elasticmapreduce:us-east-1:accountid:cluster/*", "Effect": "Allow" }, { "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": "arn:aws:events:us-east-1:accountid:rule/StepFunctionsGetEventForEMRAddJobFlowStepsRule", "Effect": "Allow" } ] } The API call in CloudTrail { "eventVersion": "1.05", "userIdentity": { "type": "AssumedRole", "principalId": "AROA36BAFPT62RGX2RMVQ:mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb", "arn": "arn:aws:sts::accountid:assumed-role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE/mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb", "accountId": "accountid", "accessKeyId": "ASIA36BAFPT6UG3XKKHI", "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "AROA36BAFPT62RGX2RMVQ", "arn": "arn:aws:iam::accountid:role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE", "accountId": "accountid", "userName": "emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE" }, "webIdFederationData": {}, "attributes": { "mfaAuthenticated": "false", "creationDate": "2020-06-08T23:29:29Z" } }, "invokedBy": "states.amazonaws.com" }, "eventTime": "2020-06-08T23:29:29Z", "eventSource": "elasticmapreduce.amazonaws.com", "eventName": "AddJobFlowSteps", "awsRegion": "us-east-1", "sourceIPAddress": "states.amazonaws.com", "userAgent": "states.amazonaws.com", "errorCode": "ValidationException", "errorMessage": "Specified job flow ID does not exist.", "requestParameters": { "jobFlowId": "j-288AS5TZY5CY7", "steps": [ { "name": "$$.Execution.Id", "actionOnFailure": "CONTINUE", "hadoopJarStep": { "jar": "command-runner.jar", "args": [] } } ] }, "responseElements": null, "requestID": "00c685e0-0d22-43ac-8fce-29bd808462bf", "eventID": "2a51a9ef-20a5-4fd7-bd33-be42cdf3088c", "eventType": "AwsApiCall", "recipientAccountId": "accountid" }

Hi, We have been trying to enable LoggingConfiguration (Express/Standard) for the state functions we are developing as outlined by the below article: https://aws.amazon.com/about-aws/whats-new/2020/02/aws-step-functions-supports-cloudwatch-logs-standard-workflows/ The configuration was working as expected over the last few months, but the same seems broken from the last week or so. When we try to add the LoggingConfiguration via CLI / Console / Cloud Formation, we get the below error: The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID:XXXXXXXXXX) Example CLI command: aws stepfunctions update-state-machine --state-machine-arn <state machine ARN> --logging-configuration "{\"level\": \"ALL\", \"includeExecutionData\": true, \"destinations\": \[{\"cloudWatchLogsLogGroup\": {\"logGroupArn\": \"<log group ARN>\"}}]}" Cloud Formation: ExampleValidatorStateMachine: Type: AWS::StepFunctions::StateMachine DependsOn: ExampleValidatorStateMachineLogGroup Properties: StateMachineName: !Sub &#39;${StackPrefix}ExampleValidator&#39; LoggingConfiguration: IncludeExecutionData: true Level: &#39;ALL&#39; Destinations: - CloudWatchLogsLogGroup: LogGroupArn: !GetAtt \[ExampleValidatorStateMachineLogGroup, Arn] DefinitionString: |- { ….. } RoleArn: !GetAtt \[ExampleValidatorStatesExecutionRole, Arn] The StateMachine has associated role that has the required IAM Policies for Logging to CloudWatch Logs as outlined by: https://docs.aws.amazon.com/step-functions/latest/dg/cw-logs.html

Hello. I'm working on porting from the 1.x to the 2.x AWS Java SDK library. Most of the porting (sns, sqs, secretsmanager, etc.) has been straightforward. However, I'm having trouble finding anything for the stepfunctions client. I've added: <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>sfn</artifactId> </dependency> but then, I'm at a loss for finding the new\[er] equivalent for the AWSStepFunctions.java class. Oddly enough, there isn't much examples or references out there for this either. The AWS Java SDK 2.x does not have any examples either. Thanks!

Let's say I define a task state that invokes a Lambda. I'm wondering if there's a way I can pass through or default a specific optional input parameter in the `Parameters` property. ``` "someState": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "someFunction", "InvocationType": "RequestResponse", "Qualifier.$": "$.environmentId", "Payload": { "environmentId.$": "$.environmentId", "someParameter.$": "$.someParameter", } } } ``` If `someParameter` is included in the input I want it to be passed through. Else I'd want one of: • Omit it. • Be able to configure a default for it. • Or if it even defaulted to `null` automatically, that'd be workable for me. Is there a way to do that? What actually happens is an error like: ``` { "error": "States.Runtime", "cause": "An error occurred while executing the state 'someState' (entered at the event id #7). The JSONPath '$.someParameter' specified for the field 'someParameter.$' could not be found in the input" } ``` Consider this scenario: I have application environments `staging` and `production` that execute the step function. `someFunction` has aliases with those names. I want `someFunction` to utilize a new request parameter that will be passed in the input to the step function. The first thing that will happen is the Lambda code deployed to the `staging` alias will be updated to utilize the new parameter and the application running in the `staging` environment will pass it to the step function. At this point the application running in the `production` environment will not have been updated to pass the new parameter to the step function. So if I can just optionally pass through the new parameter, all would be well. But in reality it would break the `production` environment. So far the best solution I can think of is to change to... ``` "Payload": "$" ``` ...to pass everything through and pick out the parameters I'm interested in in the Lambda, but it means passing unnecessary data and the state configuration no longer documents the expected parameters. Edited by: Jesse12345 on Nov 12, 2019 1:26 PM Edited by: Jesse12345 on Nov 13, 2019 6:22 AM

IHAC who's doing a low level design on their data lake. they want to use all AWS native services where possible. they have a question on ETL orchestration best practices on AWS. They were looking at Step functions but since Glue Workflow is available since Jun 2019 they were wondering which to use or a combo. Of course they are looking for the easy button. here's their primary requirements. 1. ETL orchestration - step functions vs. Glue Workflow 1. ~150 sources all sending files various times 2. Source systems have limits on concurrency that scheduling tool must support 3. example max 10 concurrent jobs for ACME source - job scheduling tool should pole and submit jobs keeping 10 active jobs but no more than 10 4. ETL jobs should be built off of parameterized template where they pass in parameters like source, table name date and the job auto builds vs. having to maintain library of jobs/scripts per source/table. want this to be dynamically built 5. Alerts on ETL processing 1. Cloudwatch alert to SNS topics to etl teams on failures 2. Cloudwatch alert to SNS for business users(loads complete) 3. etc 6. Support downstream jobs/etl example load file A & Load file B once completed for the day should launch load file C etc

Hi', Can we use AWS Step Functions as a scheduler service ? Like AWS SF starts at a particular time in a day and invokes some HTTPS/HTTP URL with some data, is this possible ? Thanks Yatan

Hi, Step Functions can be used to create ML workflows. What is the best practice to version the code creating those workflows? boto3 code in CodeCommit? Something else? Cheers Olivier

Hello, I am trying to create a step function. But unable to create it. My admin has attached the following permissions to it for IAM ROLE. { "Version": "2012-10-17", "Statement": \[ { "Effect": "Allow", "Principal": { "Service": \[ "states.us-east-1.amazonaws.com", "lambda.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] } But still I get the following error. First ERROR after loading Get Started Page: "Error There was an issue while loading activity arns, auto complete functionality may be limited." After filling out name and selecting Catch Failure blueprint and editing ARN. I clicked on Create State Machine. I get error box on code saying: " Error code: Message: Error code: Message:" Second time when I clicked on Create State Machine. I get to select IAM role and after that a final error: (On Top of the page) "Error There was an issue while loading activity arns, auto complete functionality may be limited. An error occurred while creating your state machine, please try again. Error"