By using AWS re:Post, you agree to the Terms of Use
/AWS Step Functions/

Questions tagged with AWS Step Functions

Sort by most recent
  • 1
  • 90 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

Invalid security token error when executing nested step function on Step Functions Local

Are nested step functions supported on AWS Step Functions Local? I am trying to create 2 step functions, where the outer one executes the inner one. However, when trying to execute the outer step function, getting an error: "The security token included in the request is invalid". To reproduce, use the latest `amazon/aws-stepfunctions-local:1.10.1` Docker image. Launch the container with the following command: ```sh docker run -p 8083:8083 -e AWS_DEFAULT_REGION=us-east-1 -e AWS_ACCESS_KEY_ID=TESTID -e AWS_SECRET_ACCESS_KEY=TESTKEY amazon/aws-stepfunctions-local ``` Then create a simple HelloWorld _inner_ step function in the Step Functions Local container: ```sh aws stepfunctions --endpoint-url http://localhost:8083 create-state-machine --definition "{\ \"Comment\": \"A Hello World example of the Amazon States Language using a Pass state\",\ \"StartAt\": \"HelloWorld\",\ \"States\": {\ \"HelloWorld\": {\ \"Type\": \"Pass\",\ \"End\": true\ }\ }}" --name "HelloWorld" --role-arn "arn:aws:iam::012345678901:role/DummyRole" ``` Then add a simple _outer_ step function that executes the HelloWorld one: ```sh aws stepfunctions --endpoint-url http://localhost:8083 create-state-machine --definition "{\ \"Comment\": \"OuterTestComment\",\ \"StartAt\": \"InnerInvoke\",\ \"States\": {\ \"InnerInvoke\": {\ \"Type\": \"Task\",\ \"Resource\": \"arn:aws:states:::states:startExecution\",\ \"Parameters\": {\ \"StateMachineArn\": \"arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorld\"\ },\ \"End\": true\ }\ }}" --name "HelloWorldOuter" --role-arn "arn:aws:iam::012345678901:role/DummyRole" ``` Finally, start execution of the outer Step Function: ```sh aws stepfunctions --endpoint-url http://localhost:8083 start-execution --state-machine-arn arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorldOuter ``` The execution fails with the _The security token included in the request is invalid_ error in the logs: ``` arn:aws:states:us-east-1:123456789012:execution:HelloWorldOuter:b9627a1f-55ed-41a6-9702-43ffe1cacc2c : {"Type":"TaskSubmitFailed","PreviousEventId":4,"TaskSubmitFailedEventDetails":{"ResourceType":"states","Resource":"startExecution","Error":"StepFunctions.AWSStepFunctionsException","Cause":"The security token included in the request is invalid. (Service: AWSStepFunctions; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: ad8a51c0-b8bf-42a0-a78d-a24fea0b7823; Proxy: null)"}} ``` Am I doing something wrong? Is any additional configuration necessary?
0
answers
0
votes
8
views
asked 2 days ago

Failed to convert 'Body' to string S3.InvalidContent arn:aws:states:::aws-sdk:s3:getObject step function

I am a newbie so pardon my ignorance. I am writing a very simple step function state machine that uses the AWS SDK to retrieve a file from S3. Every time I run it the task that gets the file from S3 fails with an "S3.InvalidContent" error with "Failed to convert 'Body' to string" as the cause. The full definition of my state machine is: ``` { "Comment": "A description of my state machine", "StartAt": "GetAudioFile", "States": { "GetAudioFile": { "Type": "Task", "Parameters": { "Bucket": "11123", "Key": "test.wav" }, "Resource": "arn:aws:states:::aws-sdk:s3:getObject", "End": true } } } ``` The full text of the TaskFailed event is: ``` { "resourceType": "aws-sdk:s3", "resource": "getObject", "error": "S3.InvalidContent", "cause": "Failed to convert 'Body' to string" } ``` The full text of the CloudWatch log entry with the error is: ``` { "id": "5", "type": "TaskFailed", "details": { "cause": "Failed to convert 'Body' to string", "error": "S3.InvalidContent", "resource": "getObject", "resourceType": "aws-sdk:s3" }, "previous_event_id": "4", "event_timestamp": "1651894187569", "execution_arn": "arn:aws:states:us-east-1:601423303632:execution:test:44ae6102-b544-3cfa-e186-181cdf331493" } ``` 1. What am I doing wrong? 2. How do I fix it? 3. What additional information do you need from me? 4. Most importantly, where can I find answers to these stupid questions so I don't have to post these stupid questions on re:Post again? (I have spent nearly a day scouring AWS docs and Googling without finding anything.)
1
answers
0
votes
3
views
asked 13 days ago

Many AWS Step Functions events in CloudTrail are considered "Management Events", but should be "Data Events"

We use Step Functions pretty extensively in one of our applications. I noticed higher than expected costs in CloudTrail and GuardDuty which caused me to investigate. It looks like every call to StartExecution, SendTaskHeartbeat, StartExecution, SendTaskSuccess, SendTaskFailure, etc are all considered "Management Events" inside CloudTrail. Since all of these function are normal usage of the Step Functions service, I think they should be considered "Data Events" in the same way that regular "usage" of S3, Dynamo, and Lambda API calls are handled. By being considered "management events", they are causing a large number of events (and cost) in CloudTrail, and similar with GuardDuty. Below is a typical event caused by an API call to SendTaskHeartbeat, where you can see `"managementEvent": true` and ` "eventCategory": "Management"`. I believe this should be `"managementEvent": false` and ` "eventCategory": "Data"` ``` { "eventVersion": "1.08", "userIdentity": { "type": "AssumedRole", "principalId": "xxxxxxxxxxxxxx-04fe38ef50d84dad1", "arn": "arn:aws:sts::722537357562:assumed-role/my-role-name/i-x0x4xfxex3x8xex", "accountId": "999999999999", "accessKeyId": "ASIAXXXXXXXXXXXPB", "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "AROxxxxxxxxxxxxAGI", "arn": "arn:aws:iam::999999999999:role/my-role-name", "accountId": "999999999999", "userName": "my-role-name" }, "webIdFederationData": {}, "attributes": { "creationDate": "2022-05-03T19:21:10Z", "mfaAuthenticated": "false" }, "ec2RoleDelivery": "2.0" } }, "eventTime": "2022-05-03T20:56:18Z", "eventSource": "states.amazonaws.com", "eventName": "SendTaskHeartbeat", "awsRegion": "us-east-1", "sourceIPAddress": "3.81.182.218", "userAgent": "aws-sdk-php/3.183.13 OS/Linux/5.4.0-1030-aws GuzzleHttp/6.5.5 curl/7.68.0 PHP/7.4.3", "requestParameters": { "taskToken": "AAAAKgAAAA......AqHoA+2qxXBI=" }, "responseElements": null, "requestID": "999999999-81de-40bf-8b77-7ccbf0db5fb4", "eventID": "999999999-2193-47dd-8e3d-10a5d9e6266d", "readOnly": false, "eventType": "AwsApiCall", "managementEvent": true, "recipientAccountId": "999999999999", "eventCategory": "Management", "tlsDetails": { "tlsVersion": "TLSv1.2", "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256", "clientProvidedHostHeader": "states.us-east-1.amazonaws.com" } }
0
answers
0
votes
4
views
asked 16 days ago

AWS Lake Formation: (AccessDeniedException) when calling the GetTable operation: Insufficient Lake Formation permission(s) on table

I have implemented LakeFormation on my data bucket. I have a step function in which one step consists of running a GlueJob that reads and writes to the data catalog. I have upgraded my DataLake permissions as reported [here][1]. The Service Role that runs my Step Function has a root-type policy (granted just for debugging this issue): ```yaml Statement: - Effect: "Allow" Action: - "*" Resource: - "*" ``` On lake formation the service role has: - Administrator Rights - Database Creation rights (and Grantable) - Data Location access to the entire bucket (and Grantable) - Super rights on read and write Database (and Grantable) - Super rights on ALL tables within above Databases (and Grantable). The bucket is not encrypted. But, somehow, its access to the tables is denied with the error: ``` (AccessDeniedException) when calling the GetTable operation: Insufficient Lake Formation permission(s) on table ``` What's really strange is that the Glue Job succeeds when writing to some tables, and fails on others. And there is no real substantial difference across tables: all of them are under the same S3 prefix, parquet files, partitioned on the same key. Given the abundance of permissions granted, I am really clueless about what is causing the error. Please, send help. [1]: https://docs.aws.amazon.com/lake-formation/latest/dg/upgrade-glue-lake-formation.html
0
answers
0
votes
4
views
asked 2 months ago
1
answers
0
votes
11
views
asked 2 months ago

AWS StepFunctions - SageMaker's InvokeEndpoint block throws "validation error" when fetching parameters for itself inside iterator of Map block

I have a state-machine workflow with 3 following states: [screenshot-of-my-workflow](https://i.stack.imgur.com/4xJTE.png) 1. A 'Pass' block that adds a list of strings(SageMaker endpoint names) to the original input. (*this 'Pass' will be replaced by a call to DynamoDB to fetch list in future.*) 2. Use map to call SageMaker endpoints dictated by the array(or list) from above result. 3. Send the result of above 'Map' to a Lambda function and exit the workflow. Here's the entire workflow in .asl.json, inspired from [this aws blog](https://docs.aws.amazon.com/step-functions/latest/dg/sample-map-state.html). ``` { "Comment": "A description of my state machine", "StartAt": "Pass", "States": { "Pass": { "Type": "Pass", "Next": "InvokeEndpoints", "Result": { "Endpoints": [ "sagemaker-endpoint-1", "sagemaker-endpoint-2", "sagemaker-endpoint-3" ] }, "ResultPath": "$.EndpointList" }, "InvokeEndpoints": { "Type": "Map", "Next": "Post-Processor Lambda", "Iterator": { "StartAt": "InvokeEndpoint", "States": { "InvokeEndpoint": { "Type": "Task", "End": true, "Parameters": { "Body": "$.InvocationBody", "EndpointName": "$.EndpointName" }, "Resource": "arn:aws:states:::aws-sdk:sagemakerruntime:invokeEndpoint", "ResultPath": "$.InvocationResult" } } }, "ItemsPath": "$.EndpointList.Endpoints", "MaxConcurrency": 300, "Parameters": { "InvocationBody.$": "$.body.InputData", "EndpointName.$": "$$.Map.Item.Value" }, "ResultPath": "$.InvocationResults" }, "Post-Processor Lambda": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "Payload.$": "$", "FunctionName": "arn:aws:lambda:<my-region>:<my-account-id>:function:<my-lambda-function-name>:$LATEST" }, "Retry": [ { "ErrorEquals": [ "Lambda.ServiceException", "Lambda.AWSLambdaException", "Lambda.SdkClientException" ], "IntervalSeconds": 2, "MaxAttempts": 6, "BackoffRate": 2 } ], "End": true } } } ``` As can be seen in the workflow, I am iterating over the list from the previous 'Pass' block and mapping those to iterate inside 'Map' block and trying to access the Parameters of 'Map' block inside each iteration. Iteration works fine with number of iterators, but I can't access the Parameters inside the iteration. I get this error: ``` { "resourceType": "aws-sdk:sagemakerruntime", "resource": "invokeEndpoint", "error": "SageMakerRuntime.ValidationErrorException", "cause": "1 validation error detected: Value '$.EndpointName' at 'endpointName' failed to satisfy constraint: Member must satisfy regular expression pattern: ^[a-zA-Z0-9](-*[a-zA-Z0-9])* (Service: SageMakerRuntime, Status Code: 400, Request ID: ed5cad0c-28d9-4913-853b-e5f9ac924444)" } ``` So, I presume the error is because "$.EndpointName" is not being filled with the relevant value. How do I avoid this. But, when I open the failed execution and check the InvokeEndpoint block from graph-inspector, input to that is what I expected and above JSON-Paths to fetch the parameters should work, but they don't. [screenshot-of-graph-inspector](https://i.stack.imgur.com/3gXsM.jpg) What's causing the error and How do I fix this?
1
answers
0
votes
5
views
asked 2 months ago

How to access API Parameters of a node and add them as part of it's own output json in AWS Step Functions?

Here's some part of my StepFunction: https://i.stack.imgur.com/4Jxd9.png Here's the workflow for the "Parallel" node: ``` { "Type": "Parallel", "Branches": [ { "StartAt": "InvokeEndpoint01", "States": { "InvokeEndpoint01": { "Type": "Task", "End": true, "Parameters": { "Body": "$.Input", "EndpointName": "dummy-endpoint-name1" }, "Resource": "arn:aws:states:::aws-sdk:sagemakerruntime:invokeEndpoint" } } }, { "StartAt": "InvokeEndpoint02", "States": { "InvokeEndpoint02": { "Type": "Task", "End": true, "Parameters": { "Body": "$.Input", "EndpointName": "dummy-endpoint-name2" }, "Resource": "arn:aws:states:::aws-sdk:sagemakerruntime:invokeEndpoint" } } } ], "Next": "Lambda Invoke" }, ``` I would like to access the `EndpointName` of each node inside this Parallel block and add it as one of the keys of that particular node's output, without modifying the existing output's body and other headers.(in the above json, `EndpointName` can be found for first node inside the Parallel at `$.Branches[0].States.InvokeEndpoint01.Parameters.EndpointName`) Here's output of one of the node inside the Parallel block: ``` { "Body": "{xxxx}", "ContentType": "application/json", "InvokedProductionVariant": "xxxx" } ``` and I would like to access the API Parameter and make it something like below: ``` { "Body": "{xxxx}", "ContentType": "application/json", "InvokedProductionVariant": "xxxx", "EndpointName": "dummy-endpoint-name1" } ``` How do I do this?
2
answers
1
votes
4
views
asked 2 months ago

Step Function manage/trigger Fargate task container override storage size?

Hi, We successfully trigger fargate task through step function, Something like below: * Is there anyway to define the ephemeral storage size, cpu and memory within the container override section below? * Now, when we create fargate task, we set desire_count=0, min_count=0 and we have 0 instance always on (which is good, we don't want to always on instance). But how will max_count be treated? When we have more than max_count of concurrent step function running, will some step just hanged and waiting for the finishment of certain fargate task? Or throwing error? Thank you , in advance. "UncompressGranuleECS": { "Parameters": { "LaunchType": "FARGATE", "Cluster": "${module.cluster.ecs_cluster_name}", "TaskDefinition": "${local.fargateTaskARN}", "NetworkConfiguration": { "AwsvpcConfiguration": { "Subnets": ${jsonencode(var.subnet_ids)}, "AssignPublicIp": "DISABLED" } }, "Overrides": { "ContainerOverrides": [ { "Name":"dyen-cumulus-uncompress_granule-fargate", "Command": ["java", "-Xms7500M", "-Xmx7500M", "-jar", "/home/dockeruser/build/libs/my-jarfile.jar"], "Environment":[ { "Name":"region", "Value":"us-west-2" }, { "Name":"TASK_TOKEN", "Value.$":"$$.Task.Token" }, { "Name":"CMA_MESSAGE", "Value":"myjson:[]" } ] } ] } }, "Type": "Task", "HeartbeatSeconds": 1800, "Resource": "arn:aws:states:::ecs:runTask.waitForTaskToken", "Retry": [ { "ErrorEquals": [ "States.ALL" ], "IntervalSeconds": 10, "MaxAttempts": 0 } ], "Catch": [ { "ErrorEquals": [ "States.ALL" ], "ResultPath": "$.exception", "Next": "failureStep" } ], "Next": "MissingStep" },
2
answers
0
votes
7
views
asked 2 months ago

Step functions pass input into Fargate instance :

by looking at this AWS document: https://docs.aws.amazon.com/step-functions/latest/dg/sample-project-container-task-notification.html step functions can totally instantiate/manage fargate or ECS Task. Speaking about fargate * the example put fargate as first step. suppose fargate is the 2nd step, can we get the output message from first step (larger than 8K) into fargate task through either .sync or .waitForTaskToken model? Instead of passing input parameters through fargate containerOverride's command or commands? * If above is possible, does the code inside fargate suppose to do something like : GetActivityTaskResult getActivityTaskResult = client.getActivityTask(new GetActivityTaskRequest().withActivityArn(stringActivityArn)); String taskToken = getActivityTaskResult.getTaskToken(); String taskInput = getActivityTaskResult.getInput(); * In this step function instantiate fargate model (.sync or .waitForTaskToken), does fargate instances gets created each time when step functions "calls" it? That is, we actually would like to set desire_count and min count both to zero and forget about scaling alarms or metrics. If above is true, does it respect the max_count? I guess cost-wise, it is fine with us because how many concurrent fargate running does not matter anymore. Cost is based on usage. * is ECS or fargate able to return output values to the calling step function? that is, using sendSucess or some other api to send output back to step function for next step to use (become the input of next step)?
1
answers
0
votes
58
views
asked 3 months ago

Test Step Function with CDK - Best Practice

I created a simple test function with a task that invokes lambda. ``` Function weatherToday = Function.Builder.create(this, "WeatherToday") .code(Code.fromAsset(getClass().getClassLoader().getResource("lambda/weather_today").getPath())) .runtime(Runtime.NODEJS_14_X) .architecture(Architecture.ARM_64) .handler("WeatherToday.handler") .timeout(Duration.seconds(25)) .build(); IChainable getCurrentWeather = LambdaInvoke.Builder.create(this, "Get Current Weather") .lambdaFunction(weatherToday) .retryOnServiceExceptions(true) .build(); Pass hotState = new Pass(this, "It is HOT!"); Pass coldState = new Pass(this, "It is COLD!!"); Choice isItHot = new Choice(this, "Is it Hot?") .when(Condition.numberGreaterThanEquals("$.Payload.temp_f", 90), hotState) .otherwise(coldState); IChainable chain = Chain.start(getCurrentWeather).next(isItHot); StateMachine.Builder.create(this, "IsItHotNow").definition(chain).timeout(Duration.seconds(60))// best practice .build(); } ``` Assuming this is the best practice, I wanted to write a test that checks whether the states transition or are chained as intended. When pulling in the DefinitionString for the given state function ``` "DefinitionString": { "Fn::Join": [ "", [ "{\"StartAt\":\"Get Current Weather\",\"States\":{\"Get Current Weather\":{\"Next\":\"Is it Hot?\",\"Retry\":[{\"ErrorEquals\":[\"Lambda.ServiceException\",\"Lambda.AWSLambdaException\",\"Lambda.SdkClientException\"],\"IntervalSeconds\":2,\"MaxAttempts\":6,\"BackoffRate\":2}],\"Type\":\"Task\",\"Resource\":\"arn:", { "Ref": "AWS::Partition" }, ":states:::lambda:invoke\",\"Parameters\":{\"FunctionName\":\"", { "Fn::GetAtt": [ "WeatherToday53196B63", "Arn" ] }, "\",\"Payload.$\":\"$\"}},\"Is it Hot?\":{\"Type\":\"Choice\",\"Choices\":[{\"Variable\":\"$.Payload.temp_f\",\"NumericGreaterThanEquals\":90,\"Next\":\"It is HOT!\"}],\"Default\":\"It is COLD!!\"},\"It is COLD!!\":{\"Type\":\"Pass\",\"End\":true},\"It is HOT!\":{\"Type\":\"Pass\",\"End\":true}},\"TimeoutSeconds\":60}" ] ] } ``` The above is an array of strings with inlined intrinsic ASL functions. There does not seem to be a good way to get the state definitions to resemble the typical state function definition file as json without the intrinsic functions. My thoughts are if I can get the appropriate state machine definition as a json format of ASL it will be easier to test for the expected states. Please let me know if there is a way to do this or what may be the best practice approach?
0
answers
0
votes
5
views
asked 3 months ago

How to properly use KMS in Step Functions?

I'm working on SAML identification workflows in Step Functions where SAML messages has to be signed and returned Assertion is also encrypted. I will use KMS to store two different asymmetric keys (one for sign/verify and other for encrypt/decrypt) and tried to use for example 'kms:Sign' and 'kms:Decrypt' from SF through SDK integrations meaning task ARNs 'arn:aws:states:::aws-sdk:kms:sign' and 'arn:aws:states:::aws-sdk:kms:decrypt' but can only retrieve binary data in responses, which are not Base64-encoded. That's correct based on documentation: "When you use the HTTP API or the AWS CLI, the value is Base64-encoded. Otherwise, it is not Base64-encoded." Can I somehow always return Base64-encoded response or use binary response in context of SF json payloads? I can't figure out neither. Am I correct that SF can't decode/encode Base64? I also tried proxying through API gateway (which will use HTTP API as I think) but KMS responds always with 400 because CiphertextBlob can't be null. It isn't null, value is properly visible in step "request body payload after transformations" and I also can't figure out what prevents to call KMS through API gateway. If I will use Lambda to decode Base64 from request body, call KMS operation and encode Base64 from response body, all works nicely. Except including that SDK into Lambda code will increase total latency with multiple hundreds milliseconds because cold starts are much slower with SDK imported. Can I somehow avoid those overheads coming from Lambda and use KMS straight from SF or through API gateway?
0
answers
0
votes
6
views
asked 3 months ago

Local step-function does not apply success task request until Lambda will not finishes

Hi, I am trying to send a task-success request for my 'waitForTaskToken' step, but I see that the local step-machine server is waiting for the lambda to finish, instead of going to the next step how it works now on the AWS. At this moment task-success request from AWS CLI froze. **Local step-machine server does not return a response for 'send-task-success' request when it has been sent and waiting for the lambda to finish**. Step Functions Local Version: 1.8.1 (docker image) Step-function definition: ``` { "Comment": "", "StartAt": "waitForTaskToken", "States": { "waitForTaskToken": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken", "HeartbeatSeconds": 900, "TimeoutSeconds": 900, "Parameters": { "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:delay-function", "Payload": { "Token.$": "$$.Task.Token" } }, "End": true } } } ``` lambda 'delay-function' in this case - simple NodeJS function which prints the task token and waits 130 seconds. As we can see in the logs below: 1. 2022-02-09 10:16:25.078 - Executed local step-function 2. 2022-02-09T10:16:25.092Z - Received task token 'f68313c6-89bc-420f-b335-b8369dc5064e' and waiting 130000 ms 3. 2022-02-09 10:16:50.673 - I sent a task-success request from aws cli 'aws stepfunctions --endpoint-url http://localhost:8083 send-task-success --task-output '{"test": 123}' --task-token f68313c6-89bc-420f-b335-b8369dc5064e' after which this CLI command freezed, the **server does not return a response**. 4. 2022-02-09 10:16:50.673: - local step-function server received this request but does not return the result. [ISSUE THERE] does not stop the current step with the result from 'send-task-success' request until Lambda will not finishes. 5. 2022-02-09 10:18:35.100 - Lambda return data, (Duration: 130006.11 ms). 6. 2022-02-09 10:18:35.102 - Only now the local step-function server has returned the result for 'send-task-success' request. [ISSUE THERE] for some reason only now server returns a response for 'send-task-success' request instead of returning it at the same time it was sent (as it works now on AWS) Logs: ``` stepfunctions | 2022-02-09 10:16:25.078: [200] StartExecution <= {"sdkResponseMetadata":null,"sdkHttpMetadata":null,"executionArn":"arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273","startDate":1644401785077} stepfunctions | 2022-02-09 10:16:25.080: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"TaskScheduled","PreviousEventId":2,"TaskScheduledEventDetails":{"ResourceType":"lambda","Resource":"invoke.waitForTaskToken","Region":"us-east-1","Parameters":"{\"FunctionName\":\"arn:aws:lambda:us-east-1:123456789012:function:function\",\"Payload\":{\"Token\":\"f68313c6-89bc-420f-b335-b8369dc5064e\"}}","TimeoutInSeconds":900,"HeartbeatInSeconds":900}} stepfunctions | 2022-02-09 10:16:25.081: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"TaskStarted","PreviousEventId":3,"TaskStartedEventDetails":{"ResourceType":"lambda","Resource":"invoke.waitForTaskToken"}} lambda | START RequestId: b7e0b919-6519-4f84-bb36-70c540c8d550 Version: $LATEST lambda | 2022-02-09T10:16:25.092Z b7e0b919-6519-4f84-bb36-70c540c8d550 INFO >>> f68313c6-89bc-420f-b335-b8369dc5064e stepfunctions | 2022-02-09 10:16:50.673: SendTaskSuccess => {"requestClientOptions":{"readLimit":131073,"skipAppendUriPath":false},"requestMetricCollector":null,"customRequestHeaders":null,"customQueryParameters":null,"cloneSource":null,"sdkRequestTimeout":null,"sdkClientExecutionTimeout":null,"taskToken":"f68313c6-89bc-420f-b335-b8369dc5064e","output":"{\"test\": 123}","generalProgressListener":{"syncCallSafe":true},"readLimit":131073,"cloneRoot":null,"requestCredentials":null,"requestCredentialsProvider":null} stepfunctions | 2022-02-09 10:17:51.380: SendTaskSuccess => {"requestClientOptions":{"readLimit":131073,"skipAppendUriPath":false},"requestMetricCollector":null,"customRequestHeaders":null,"customQueryParameters":null,"cloneSource":null,"sdkRequestTimeout":null,"sdkClientExecutionTimeout":null,"taskToken":"f68313c6-89bc-420f-b335-b8369dc5064e","output":"{\"test\": 123}","generalProgressListener":{"syncCallSafe":true},"readLimit":131073,"cloneRoot":null,"requestCredentials":null,"requestCredentialsProvider":null} } "Token": "f68313c6-89bc-420f-b335-b8369dc5064e"7e0b919-6519-4f84-bb36-70c540c8d550 INFO EVENT: lambda | END RequestId: b7e0b919-6519-4f84-bb36-70c540c8d550 lambda | REPORT RequestId: b7e0b919-6519-4f84-bb36-70c540c8d550 Duration: 130006.11 ms Billed Duration: 130007 ms Memory Size: 3008 MB Max Memory Used: 3008 MB stepfunctions | 2022-02-09 10:18:35.100: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"TaskSubmitted","PreviousEventId":4,"TaskSubmittedEventDetails":{"ResourceType":"lambda","Resource":"invoke.waitForTaskToken","Output":"{\"StatusCode\":200,\"FunctionError\":null,\"LogResult\":null,\"Payload\":\"$LATEST\",\"ExecutedVersion\":null}"}} stepfunctions | 2022-02-09 10:18:35.101: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"TaskSucceeded","PreviousEventId":5,"TaskSucceededEventDetails":{"ResourceType":"lambda","Resource":"invoke.waitForTaskToken","Output":"{\"test\": 123}"}} stepfunctions | 2022-02-09 10:18:35.102: [200] SendTaskSuccess <= {"sdkResponseMetadata":null,"sdkHttpMetadata":null} stepfunctions | 2022-02-09 10:18:35.102: [200] SendTaskSuccess <= {"sdkResponseMetadata":null,"sdkHttpMetadata":null} stepfunctions | 2022-02-09 10:18:35.103: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"TaskStateExited","PreviousEventId":6,"StateExitedEventDetails":{"Name":"waitForTaskToken","Output":"{\"test\": 123}"}} stepfunctions | 2022-02-09 10:18:35.124: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:e2e402fa-6bbd-4b56-87f1-4cf1a8fc1273 : {"Type":"ExecutionSucceeded","PreviousEventId":7,"ExecutionSucceededEventDetails":{"Output":"{\"test\": 123}"}} ```
1
answers
0
votes
10
views
asked 3 months ago

AWS: Set step function timeout

I am new to AWS. We have a yaml file defined like below to run our step function. I want to set a timeout for the step function in AWS parameter store and refer it in the file. I created a parameter from AWS console. But I am not sure how to refer it in the yaml file. After reading the documentation I understood that it is declared under "States" property as "Timeout". xxxStateMachine: Type: AWS::StepFunctions::StateMachine Properties: StateMachineName: !Sub ${AWS::StackName}-StateMachine RoleArn: !Sub arn:aws:iam::${AWS::AccountId}:role/roleName DefinitionString: !Sub - |- { "Comment": "Calls xxx state update job invoker lambda", "StartAt": "xxxCheckJob", "States": { "Timeout": ${}, // IS this where the timeout has to be defined? "xxxCheckJob": { "Type": "Task", "Resource": "${lambdaBDCArn}", "ResultPath": "$.isBusinessDay", "OutputPath": "$", "Next": "xxxCheckJobStatus" }, "businessDayCheckJobStatus": { "Type": "Choice", "Choices": [ { "Variable": "$.x", "BooleanEquals": true, "Next": "xxxStatexxJob" }, { "Not": { "Variable": "$.isBuxxy", "BooleanEquals": true }, "Next": "SuccessState" } ] }, "xxxStateUpdateJob": { "Type": "Task", "Resource": "${lambdaRSUArn}", "ResultPath": "$.detail.requestParameters.invocationStatus", "OutputPath": "$.detail.requestParameters", "Next": "xxxStateUpdateJobStatus" }, "xxxStateUpdateJobStatus": { "Type": "Task", "Resource": "${lambdaJSArn}", "Parameters": { "jobName.$": "$.invocationStatus.jobId" }, "ResultPath": "$.jobStatus", "OutputPath": "$", "Next": "checkJobStatus" }, "checkJobStatus": { "Type": "Choice", "Choices": [ { "Variable": "$.jobStatus.status", "StringEquals": "FAILED", "Next": "FailState" }, { "Variable": "$.jobStatus.status", "StringEquals": "SUCCEEDED", "Next": "SuccessState" }, { "Not": { "Variable": "$.jobStatus.status", "StringEquals": "SUCCEEDED" }, "Next": "Wait X Seconds" } ] }, "Wait X Seconds": { "Type": "Wait", "Seconds": 20, "Next": "xxxStateUpdateJobStatus" }, "SuccessState": { "Type": "Succeed" }, "FailState": { "Type": "Fail", "Cause": "Invalid response.", "Error": "ErrorA" } } } - {lambdaRSUArn: !GetAtt [ xx, Arn ], lambdaJSArn: !GetAtt [ xx, Arn ], lambdaBDCArn: !GetAtt [ xx, Arn ]} I have the following questions: 1. How to access timeout value in the yaml file. Is this the syntax - "Timeout": ${parameterName} 2. How to configure the step function so that it exits if the timeout reaches and job status is in pending state. 3. How to configure the step function so that it does not exits if the timeout reaches and the job status is in running state. Can anyone help me configure these?
1
answers
0
votes
12
views
asked 4 months ago

Coordinating Step functions from App Flow -> Event Bridge -> DynamoDB

Having a real hard time getting data into Dynamo using Step Functions. The data flow is: 1. App Flow from Salesforce to S3 2. Event Bridge watching the S3 bucket 3. Map the data from S3 and store in Dynamo. This _almost_ works. Whenever App Flow gets new data from SalesForce, the step function is invoked. After the data is pulled from S3 (using GetItem), the next step is a Lambda in C#. This Lambda just deserializes the JSONL, populates a C# model for each item, and returns an array of the model. Next step is a Map step that passes each item in the Array to DynamoDB PutItem. Everything works, until you try to write a Boolean. Strings and Numbers are written fine, but it blows up if it sees a boolean. DynamoDB won't convert the boolean value. Things I have tried that didn't work: 1. Write the entire model object to Dynamo - blows up on bool values 2. Update PutItem config to map each field - blows up if there are any Nulls (nulls get dropped when the return value from the Lambda is serialized into JSON) 3. Serializing the values back into a JSON string inside the Lambda - blows up on bool values 4. Returning a list of DynamoDB Documents - blows up because each item in the document gets serialized as an empty object 5. Bypassing the Lambda altogether and try passing the JSONL to the Map step - blows up because it's not an array When trying write Bool values, the error is "Cannot construct instance of `com.amazonaws.services.dynamodbv2.model.AttributeValue` (although at least one Creator exists): no boolean/Boolean-argument constructor/factory method to deserialize from boolean value (false)" I can't see any obvious way to get this working, unless we convert _every_ item to a string, which causes problems reading the model back out later, as we lose type information and makes deserializing the model difficult.
5
answers
0
votes
16
views
asked 4 months ago

jsii.errors.JSIIError: Cannot read properties of undefined (reading 'bindToGraph')

HI All This is my first implementation of StateMachineFragment. Goal: Attempting to create a class for re-usable lambda state. This class can take a parameter and pass this as payload to Lambda and the lambda will execute the right query based on the payload. Below is my POC code to 'classs-ify' the lambda and the call to statemachine. ``` from aws_cdk import ( Duration, Stack, # aws_sqs as sqs, aws_stepfunctions as _stepfunctions, aws_stepfunctions as sfn, aws_stepfunctions_tasks as _stepfunctions_tasks, aws_lambda as _lambda, ) from constructs import Construct class SubMachine(_stepfunctions.StateMachineFragment): def __init__(self, parent, id, *, jobTypeParam): super().__init__(parent, id) existingFunc = _lambda.Function.from_function_arn(self, "ExistingLambdaFunc", function_arn="arn:aws:lambda:us-east-1:958$#$#$#$:function:dummyFunction") lambda_invoked = _stepfunctions_tasks.LambdaInvoke(self, "someID", lambda_function=existingFunc) wait_10_seconds = _stepfunctions.Wait(self, "Wait for 10 seconds", time=_stepfunctions.WaitTime.duration(Duration.seconds(10)) ) self._start_state = wait_10_seconds self._end_states = [lambda_invoked.end_states] def start_state(self): return self._start_state def end_states(self): return self._end_states class StepfunctionsClasStack(Stack): def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None: super().__init__(scope, construct_id, **kwargs) test_lambda_1 = SubMachine(self, "SubMachine1", jobTypeParam="one") state_machine = _stepfunctions.StateMachine(self, "TestStateMachine", definition=test_lambda_1, # role=marketo_role ) ``` When I try and deploy this code, I get the following error: ``` jsii.errors.JSIIError: Cannot read properties of undefined (reading 'bindToGraph') ``` I am not sure where I am going wrong. Thoughts? Thanks
1
answers
0
votes
11
views
asked 5 months ago

Start_state and End_states for step functions

Hi All All I am trying to do is to create a reusable ambda component where I can pass parameters to the class so that the Lambda can do different things, based on input param. I am using CDK in python to deploy the stack. I would like to create a parallel stepfunction, where I can pass the same lambda using different param/payload so they can be branches. I am running the following code: ``` from aws_cdk import ( # Duration, Stack, # aws_sqs as sqs, aws_stepfunctions as _stepfunctions, aws_stepfunctions_tasks as _stepfunctions_tasks, aws_lambda as _lambda, ) from constructs import Construct class LambdaJob(_stepfunctions.StateMachineFragment): def __init__(self, parent, id, *, jobTypeParam): super().__init__(parent, id) existingFunc = _lambda.Function.from_function_arn(self, "ExistingLambdaFunc", function_arn="arn:aws:lambda:us-east-1:95842$$$$$:function:dummyFunction") lambda_invoked = _stepfunctions_tasks.LambdaInvoke(self, "someID", lambda_function=existingFunc) wait_10_seconds = _stepfunctions.Wait(self, "Wait for 10 seconds", time=_stepfunctions.WaitTime.duration(Duration.seconds(10)) ) self.start_state = wait_10_seconds class StepfunctionsClasStack(Stack): def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None: super().__init__(scope, construct_id, **kwargs) test_lambda_1 = LambdaJob(self, "Quick") #state_machine = _stepfunctions.StateMachine(self, "TestStateMachine", # definition=LambdaJob(self, "Quick"), # # role=marketo_role # ) ``` However i keep getting the following error: ``` TypeError: Can't instantiate abstract class LambdaJob with abstract methods end_states, start_state ``` Any thoughts on what I am doing wrong ? Thanks
1
answers
0
votes
9
views
asked 5 months ago

Design suggestions

Hello All I am expanding the scope of the project, and wanted some suggestions/comments on if the right tech stack is being used. (Job - Pulling Leads and Activities from Marketo). Each job has a different query. **Scope**: We need 2 jobs to be run daily, however here is the catch. Each job should be created and queued first. Once done, we can poll to see if the job on Marketo side is completed and the file is downloaded. The file download can take more than 15 mins. THe file an be downloaded using the Job Id, using the 2 jobs that were created earlier. **Current/Implemented Solution**: I started with solving for Leads only and here is the stack that was worked out. The Event will be triggered, on a daily basis using event bridge. The task that is to be triggered is a step function. The Sfn, first calls a Lambda to create job, waits for 1 min, another lambda to queue the job, then wait for 10 mins and 3rd lambda to check status of file. If not ready, wait for 10 and poll again for file status(this is a loop with choice to keep checking file status). Once file is ready, call a container(fargate/ECS) and pass the Job Id as containerOverride to the container. Run the job on container to download the file and upload the file to S3. **Incorporate pulling activities into the above flow**: Since the queuing and Polling(for file status) lamba are similar, and the first lambda(creating the job) is different, I though of creating a parallel task where each branch does create, queue, poll and download the file(using the implemented solution, so 2 lambdas for job create and reuse the 2nd and 3rd lambdas). Once the complete chain(one for Leads and one for activities) is done, have a consolidation stage where the output from each container output is collected and an SNS message of job completion is send. I am looking forward to your suggestions to see if the suggested above workflow is how it should be done or is there any other technology that I should use. **Design Constraints**: I need to create and queue all the jobs first before starting the downloading since Marketo has a limit of 500mb for file download. Hence I the need to create and queue all the jobs first , and then only start the job to download of files. Thanks for your suggestions. Regards Tanmay
1
answers
0
votes
34
views
asked 5 months ago

Error, AWS Step Function DynamoDB Put Item, Map and List inside Map

Hello, I'm having troubles inserting an item in DynamoDB with this Structure from Step Functions. ``` { "id":{ "S":"" }, "state_machine_execution_id":{ "S":"" }, "timestamp":{ "S":"" }, "map_param1":{ "M":{ "sub_map1":{ "M":{ } }, "sub_map2":{ "S":"" }, "sub_map3":{ "M":{ } }, "sub_mapN":{ "M":{ } }, "List_1":{ "L":[ ] }, "List_2":{ "L":[ ] } } } } ``` This is the statement that I am trying to make it work without success ``` { "DynamoDB Save Item":{ "Type":"Task", "Resource":"arn:aws:states:::dynamodb:putItem", "Parameters":{ "TableName.$":"$.TableName", "Item":{ "id.$":"$.id", "state_machine_execution_id.$":"$$.Execution.Id", "timestamp.$":"$.timestamp", "map_param1":{ "M":{ "M.$":"$.var_for_map_params", "List_1":{ "L.$":"$.List_1" }, "List_2":{ "L.$":"$.List_2" } } } } } } } ``` I'm getting this error ``` The Parameters "{A lot of data flow} "could not be used to start the Task: [The field \"foo\" is not supported by Step Functions]" ``` This "foo" field, has nothing to do, if make changes to the Parameters: map_param1 statement in Step Function works example This works ok ``` "map_param1":{ "M.$":"$.var_for_map_params" } ``` And This works ok too ``` "map_param1":{ "M":{ "List_1":{ "L.$":"$.List_1" }, "List_2":{ "L.$":"$.List_2" } } } ``` But not this (the combination) ``` "map_param1":{ "M":{ "M.$":"$.var_for_map_params", "List_1":{ "L.$":"$.List_1" }, "List_2":{ "L.$":"$.List_2" } } } ``` Can someone help me? thanks a lot Regards
1
answers
0
votes
21
views
asked 5 months ago

Write containerOverwrites from Lambda to Container

Hello All I have a step function that is calling a lambda, which does some processing, and gets the id of the job that needs to be processed by the ecs runtask. What I am trying to do is to pass the job Id as containerOverride so that each run a different Id can be passed to the ecs Run task. Here is the dummy lambda Output: ``` Test Event Name dummyTest Response { "containerOverrides": " [ { \"name\": \"getFileTask\", \"environment\": [ { \"name\": \"name1\", \"value\": \"123\" }, { \"name\": \"DATE\", \"value\": \"1234-12-12\" }, { \"name\": \"SCRIPT\", \"value\": \"123456\" } ] } ] " } ``` Dummy Lambda Code: ``` def lambda_handler(event, context): # TODO implement print("Success") overridden_Text = ' [ { "name": "getFileTask", "environment": [ { "name": "name1", "value": "123" }, { "name": "DATE", "value": "1234-12-12" }, { "name": "SCRIPT", "value": "123456" } ] } ] ' return{ 'containerOverrides': overridden_Text } ``` Here is the record when the ECS run task is triggered(TaskStateEntered) from the step function: ``` { "name": "ECS RunTask", "input": { "containerOverrides": " [ { \"name\": \"getFileTask\", \"environment\": [ { \"name\": \"name1\", \"value\": \"123\" }, { \"name\": \"DATE\", \"value\": \"1234-12-12\" }, { \"name\": \"SCRIPT\", \"value\": \"123456\" } ] } ] " }, "inputDetails": { "truncated": false } } ``` The issue is that when the run task enters the TaskSubmitted stage: ``` "LastStatus": "PROVISIONING", "LaunchType": "FARGATE", "Memory": "4096", "Overrides": { "ContainerOverrides": [ { "Command": [], "Environment": [], "EnvironmentFiles": [], "Name": "getFileTask", "ResourceRequirements": [] } ], "InferenceAcceleratorOverrides": [] }, "PlatformFamily": "Linux", "PlatformVersion": "1.4.0", "StartedBy": "AWS Step Functions", ``` For what ever reasons the Environment variable is not being pushed from lambda output to the Container as launch override. Is there something that I am doing incorrectly ? Thanks
1
answers
0
votes
11
views
asked 5 months ago

Why aren't the HTTP headers passed from API Gateway to Step Functions?

I have an endpoint set up on API Gateway with integration to a step function - the integration is working well and my function is executed. However I have a need to access the headers on the initial request to the api gateway (as they need to be passed on to an API call made by one of the steps in the step function) I've added the http headers to the API Gateway Method Request and also done this in the HTTP Headers section of the Integration Request, then in the mapping template I have ``` #set($inputbody = $input.json('$')) { "method": "$context.httpMethod", "input": "$util.escapeJavaScript($inputbody)", "stateMachineArn": "MyStateMachineARN", "headers": { #foreach($param in $input.params().header.keySet()) "$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end #end } } ``` When I test this I see the headers in the logs after the request body has been transformed - before it executes the step function ``` request body after transformations: { "method": "POST", "input": "{\"surname\":\"TESTSURNAME\"}", "stateMachineArn": "MyStateMachineARN", "headers": { "HeaderA": "ValueA" , "HeaderB": "ValueB" } } ``` But in the step functions I am struggling to see the headers - the input I can see at the start of the execution is only ``` { "surname::"TESTSURNAME" } ``` I have inputPath set to $ and the same for the payload. All the suggestions I've found online point to the mapping template but I can not get it to work - any ideas what I'm doing wrong?
1
answers
0
votes
18
views
asked 5 months ago
3
answers
0
votes
24
views
asked 5 months ago

AWS Step functions Api Gateway integration RequestBody.$ bug

I have a state machine splitted in some different lambdas to complete a process. One of the steps is a call to apigateway lambda function. Previous call is a call to lambda function and in ASL language I tell to pass the object overrided by this function to the next step. When it comes to Apigateway step, PathParameters are filled ok with the jsonpath accessors but when I try to use in RequestBody the received value are always empty. I'm using golang language State Machine "UploadFile": { "Type": "Task", "Resource": "${UploadFileFunctionArn}", "Retry": \[ { "ErrorEquals": \[ "States.TaskFailed" ], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0 } ], "Catch": \[ { "ErrorEquals": \[ "States.ALL" ], "ResultPath": null, "Next": "Fail" } ], "ResultPath": "$", "Next": "UpdateTerminal" }, "UpdateTerminal": { "Type": "Task", "Next": "HasBeenUpdated", "Resource": "${UpdateTerminalFunctionArn}", "Parameters": { "ApiEndpoint": "${APIDomainName}/${APIBasePath}", "Method": "POST", "Stage": "${ApiStageName}", "Path": "", "PathParameters": { "terminalId.$": "$.terminalId" }, "RequestBody.$": "$", "AuthType": "NO_AUTH" }, "Retry": \[ { "ErrorEquals": \[ "States.TaskFailed" ], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0 } ] }, Update terminal lambda logs : fmt.Println("request.Body", request.Body) fmt.Println("request.PathParameters\['terminalId']", request.PathParameters\["terminalId"]) 2020-12-07T13:55:23.889+01:00 request.Body 2020-12-07T13:55:23.889+01:00 request.PathParameters\['terminalId'] dc725a3a-0b96-4b14-8036-5c36514466556 Edited by: eguzkilorebeltza on Dec 7, 2020 5:08 AM
1
answers
0
votes
2
views
asked a year ago

How do I use JSON Lambda output in Step Functions

I have a (Python) Lambda function which returns a JSON object. I'd like to join that object on to my Step Functions state, and access its sub-fields in the next state... Something like: ```python # Lambda Function def handler(event, context): return json.dumps({ "ThingARN": "...", "AnotherField": "..." }) ``` ```json { "StartAt": "LambdaStuff", "States": { "LambdaStuff": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "MyCoolLambdaFunction", "Payload.$": "$" }, "ResultPath": "$.LambdaResult", "Next": "NextActivity" }, "NextActivity": { "Type": "Task", "Resource": "...", "Parameters": { "ARN.$": "$.LambdaResult.ThingARN" }, "ResultPath": "$.AnotherResult", "End": true } } } ``` ...But my state machine is failing (cancelling the `NextActivity` task) because what gets populated to `$.LambdaResult` is a big object with Lambda metadata and the ***stringified*** result payload - like this: ```json { "ExecutedVersion": "$LATEST", "Payload": "{\"ThingARN\": \"...\", \"AnotherField\": \"...\"}", "SdkHttpMetadata": { "HttpHeaders": { ... }, // All kinds'a stuff "HttpStatusCode": 200 }, "SdkResponseMetadata": { "RequestId": "..." }, "StatusCode": 200 } ``` [This docs page][1] seems to imply that what I'm trying to do is possible, but the example is clearly very simplified/stripped-down. All the other worked examples I've found (e.g. ["InputPath, OutputPath and ResultPath Example"][2]) just seem to have Lambdas returning simple strings, rather than structured objects. I guess I'm missing something obvious? [1]: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-state-machine-data.html#concepts-state-machine-data-state-input-output [2]: https://docs.amazonaws.cn/en_us/step-functions/latest/dg/input-output-example.html
1
answers
0
votes
104
views
EXPERT
asked 2 years ago

AWS Step Function does not recognize job flow ID

I have created trivial step function to add a step to an EMR cluster. The jobflowid is not being recognised. Does anyone have a working solution here? Or pointer to what set up may be missing? The state machine: { "StartAt": "add_emr_step", "States": { "add_emr_step": { "End": true, "InputPath": "$", "Parameters": { "ClusterId": "j-INTERESTINGID", "Step": { "Name": "$$.Execution.Id", "ActionOnFailure": "CONTINUE", "HadoopJarStep": { "Jar": "command-runner.jar", "Args": [ "spark-submit --deploy-mode cluster s3://myfavouritebucketname86fda855-nhjwrxp55888/scripts/csv_to_parquet.py" ] } } }, "Type": "Task", "Resource": "arn:aws:states:::elasticmapreduce:addStep.sync", "ResultPath": "$.guid" } }, "TimeoutSeconds": 30 } Even with admin permissions the following error is returning. The EMR cluster is marked as visible. Copying and pasting the jobflowid into the console, it shows as expected. > Error > > EMR.AmazonElasticMapReduceException > > Cause > > Specified job flow ID does not exist. (Service: > AmazonElasticMapReduce; Status Code: 400; Error Code: > ValidationException; Request ID: be70a470-ef6d-4356-98cd-0fc36d2cd132) The IAM Role: { "Version": "2012-10-17", "Statement": [ { "Action": [ "elasticmapreduce:AddJobFlowSteps", "elasticmapreduce:DescribeStep", "elasticmapreduce:CancelSteps" ], "Resource": "arn:aws:elasticmapreduce:us-east-1:accountid:cluster/*", "Effect": "Allow" }, { "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": "arn:aws:events:us-east-1:accountid:rule/StepFunctionsGetEventForEMRAddJobFlowStepsRule", "Effect": "Allow" } ] } The API call in CloudTrail { "eventVersion": "1.05", "userIdentity": { "type": "AssumedRole", "principalId": "AROA36BAFPT62RGX2RMVQ:mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb", "arn": "arn:aws:sts::accountid:assumed-role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE/mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb", "accountId": "accountid", "accessKeyId": "ASIA36BAFPT6UG3XKKHI", "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "AROA36BAFPT62RGX2RMVQ", "arn": "arn:aws:iam::accountid:role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE", "accountId": "accountid", "userName": "emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE" }, "webIdFederationData": {}, "attributes": { "mfaAuthenticated": "false", "creationDate": "2020-06-08T23:29:29Z" } }, "invokedBy": "states.amazonaws.com" }, "eventTime": "2020-06-08T23:29:29Z", "eventSource": "elasticmapreduce.amazonaws.com", "eventName": "AddJobFlowSteps", "awsRegion": "us-east-1", "sourceIPAddress": "states.amazonaws.com", "userAgent": "states.amazonaws.com", "errorCode": "ValidationException", "errorMessage": "Specified job flow ID does not exist.", "requestParameters": { "jobFlowId": "j-288AS5TZY5CY7", "steps": [ { "name": "$$.Execution.Id", "actionOnFailure": "CONTINUE", "hadoopJarStep": { "jar": "command-runner.jar", "args": [] } } ] }, "responseElements": null, "requestID": "00c685e0-0d22-43ac-8fce-29bd808462bf", "eventID": "2a51a9ef-20a5-4fd7-bd33-be42cdf3088c", "eventType": "AwsApiCall", "recipientAccountId": "accountid" }
1
answers
0
votes
14
views
asked 2 years ago

Bug with LoggingConfiguration

Hi, We have been trying to enable LoggingConfiguration (Express/Standard) for the state functions we are developing as outlined by the below article: https://aws.amazon.com/about-aws/whats-new/2020/02/aws-step-functions-supports-cloudwatch-logs-standard-workflows/ The configuration was working as expected over the last few months, but the same seems broken from the last week or so. When we try to add the LoggingConfiguration via CLI / Console / Cloud Formation, we get the below error: The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID:XXXXXXXXXX) Example CLI command: aws stepfunctions update-state-machine --state-machine-arn <state machine ARN> --logging-configuration "{\"level\": \"ALL\", \"includeExecutionData\": true, \"destinations\": \[{\"cloudWatchLogsLogGroup\": {\"logGroupArn\": \"<log group ARN>\"}}]}" Cloud Formation: ExampleValidatorStateMachine: Type: AWS::StepFunctions::StateMachine DependsOn: ExampleValidatorStateMachineLogGroup Properties: StateMachineName: !Sub &#39;${StackPrefix}ExampleValidator&#39; LoggingConfiguration: IncludeExecutionData: true Level: &#39;ALL&#39; Destinations: - CloudWatchLogsLogGroup: LogGroupArn: !GetAtt \[ExampleValidatorStateMachineLogGroup, Arn] DefinitionString: |- { ….. } RoleArn: !GetAtt \[ExampleValidatorStatesExecutionRole, Arn] The StateMachine has associated role that has the required IAM Policies for Logging to CloudWatch Logs as outlined by: https://docs.aws.amazon.com/step-functions/latest/dg/cw-logs.html
8
answers
0
votes
91
views
asked 2 years ago

Is there a way to pass through specific optional input parameters

Let's say I define a task state that invokes a Lambda. I'm wondering if there's a way I can pass through or default a specific optional input parameter in the `Parameters` property. ``` "someState": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "someFunction", "InvocationType": "RequestResponse", "Qualifier.$": "$.environmentId", "Payload": { "environmentId.$": "$.environmentId", "someParameter.$": "$.someParameter", } } } ``` If `someParameter` is included in the input I want it to be passed through. Else I'd want one of: • Omit it. • Be able to configure a default for it. • Or if it even defaulted to `null` automatically, that'd be workable for me. Is there a way to do that? What actually happens is an error like: ``` { "error": "States.Runtime", "cause": "An error occurred while executing the state 'someState' (entered at the event id #7). The JSONPath '$.someParameter' specified for the field 'someParameter.$' could not be found in the input" } ``` Consider this scenario: I have application environments `staging` and `production` that execute the step function. `someFunction` has aliases with those names. I want `someFunction` to utilize a new request parameter that will be passed in the input to the step function. The first thing that will happen is the Lambda code deployed to the `staging` alias will be updated to utilize the new parameter and the application running in the `staging` environment will pass it to the step function. At this point the application running in the `production` environment will not have been updated to pass the new parameter to the step function. So if I can just optionally pass through the new parameter, all would be well. But in reality it would break the `production` environment. So far the best solution I can think of is to change to... ``` "Payload": "$" ``` ...to pass everything through and pick out the parameters I'm interested in in the Lambda, but it means passing unnecessary data and the state configuration no longer documents the expected parameters. Edited by: Jesse12345 on Nov 12, 2019 1:26 PM Edited by: Jesse12345 on Nov 13, 2019 6:22 AM
2
answers
0
votes
26
views
asked 3 years ago
  • 1
  • 90 / page