Error with AWS step function input parameters

Yep, this issue happens a lot with Step Functions when you're trying to increment a counter and accidentally wrap it inside another object. What you're seeing: "iteration_count": { "iteration_count": 1 } is because the Pass state is nesting the result by creating a new object inside the existing iteration_count key.

The fix is simple and works: Instead of writing the result like this:

"ResultPath": "$.iteration_count", "Parameters": { "iteration_count.$": "States.MathAdd($.iteration_count, 1)" } You just want to replace the whole iteration_count field directly with the new value. Do this instead:

"ResultPath": "$.iteration_count", "Parameters": { "Value.$": "States.MathAdd($.iteration_count, 1)" }

That’ll give you:

"iteration_count": 2 If you want to avoid that "Value" wrapper too, another clean option is to just do:

"ResultPath": "$.iteration_count", "Parameters": { "iteration_count.$": "States.MathAdd($.iteration_count.iteration_count, 1)" }

=================================================== Like, Replace just the iteration_count field (cleanway):

"increment_counter": { "Type": "Pass", "ResultPath": "$.iteration_count", "Parameters": { "iteration_count.$": "States.MathAdd($.iteration_count, 1)" }, "Next": "continue_loop" } This will ensure that the value under $.iteration_count is directly updated to a number (e.g., 2, 3, etc.), not an object.

======================================================

This is what I am thinking for now:

1. Avoid indefinite looping in a single execution. Even though Standard Step Functions can run for up to 1 year, each execution is capped at 25,000 state transitions.

Once that limit is hit, your workflow will fail with a HistoryLimitExceeded error, and debugging gets harder as the execution grows in size.

2. Use loop batching + handoff after N iterations. Your idea of ending the current execution after, say, 100 loops, then starting a new Step Function with the same input, is a solid pattern.

This keeps the execution history manageable and gives you better observability and control.

3. Trigger new executions cleanly You can trigger the next batch using:

A Lambda inside your Step Function using StartExecution

Or use an EventBridge rule if you prefer separating orchestration logic

Track the iteration count in your state input (e.g., iterationCount) so you know when to restart.

Further, you can do like: Add an iteration counter to your state input: "iterationCount": 0 In your loop logic, increment the count and include a Choice state to:

Continue loop if under threshold (e.g., < 100)

Trigger StartExecution with the same input (or updated input) if the threshold is reached.

Use "End" to gracefully close out the current execution.

Nice, sounds like your loop is coming together well now.

Regarding the states:StartExecution error ,that one’s pretty common when chaining Step Function executions from within another Step Function.

The error means the IAM role that your current Step Function is running under (in this case, the role dev-chan-insights-data-insights-data-pipeline) doesn’t have permission to start another execution of the same Step Function, or any other, depending on your config.

In your IAM policy attached to that role, make sure you’ve added this permission: { "Effect": "Allow", "Action": "states:StartExecution", "Resource": "arn:aws:states:us-west-2:952817440625:stateMachine:dev-chan-insights-data-pipeline" } If you want to allow it to start any Step Function in your account (e.g., for chaining different workflows), you can loosen it slightly:

"Resource": "arn:aws:states:us-west-2:952817440625:stateMachine:*"

Where to add it: Either in your role’s inline policy Or in the step_function_execution_policy.tf file (assuming you're using Terraform) Make sure the role assumed by the Step Function (the one shown in the error) actually has this policy attached sometimes the role is created separately from the state machine.

Additionally, I'm thinking, but please cross-check once. If you're passing input, don't forget to add the Input.$ field in your startExecution step too, like: "Parameters": { "StateMachineArn.$": "$.step_function_arn", "Input.$": "$$.Execution.Input" } That’ll make sure the new execution gets the same starting payload.

The error you're encountering in the second iteration of your Step Function, specifically at the source_tests step, appears to be related to a data type mismatch in the input parameters. The error message suggests that the system is expecting an array of strings for a certain parameter, but it's receiving an object instead.

This issue likely stems from how the data is being passed between states in your Step Function. After the first iteration, it seems that the structure of your input data is changing, causing the source_tests step to receive an unexpected data type.

To resolve this, you should:

1. Check the output of the previous step (placement_post_transformation_tests or continue_loop) to ensure it's maintaining the correct data structure.

2. Verify that the source_tests step is receiving the expected input format. You might need to add a Pass state before source_tests to reshape the data if necessary.

3. Ensure that the "Command. The error you're encountering in the second iteration of your Step Function, specifically at the source_tests step, appears to be related to a data type mismatch in the input parameters. The error message suggests that the system is expecting an array of strings for a certain parameter, but it's receiving an object instead.

This issue likely stems from how the data is being passed between states in your Step Function. After the first iteration, it seems that the structure of your input data is changing, causing the source_tests step to receive an unexpected data type.

To resolve this, you should:

1. Check the output of the previous step (placement_post_transformation_tests or continue_loop) to ensure it's maintaining the correct data structure.

2. Verify that the source_tests step is receiving the expected input format. You might need to add a Pass state before source_tests to reshape the data if necessary.

3. Ensure that the "Command.$" parameter in the source_tests task is correctly referencing an array of strings. You might need to adjust the JsonPath expression to select the correct part of your state data.

4. Consider adding more robust error handling and data validation at each step to catch and handle unexpected data structures.

5. Use the Step Functions console to examine the exact input and output of each state during execution. This can help pinpoint where the data structure is changing unexpectedly.

6. If the data structure is intentionally changing between iterations, you may need to add logic to handle different input formats in your source_tests task.

Remember, when working with Step Functions, it's crucial to ensure that the output of each state matches the expected input of the next state, especially when dealing with loops or multiple iterations. quot; parameter in the source_tests task is correctly referencing an array of strings. You might need to adjust the JsonPath expression to select the correct part of your state data.

4. Consider adding more robust error handling and data validation at each step to catch and handle unexpected data structures.

5. Use the Step Functions console to examine the exact input and output of each state during execution. This can help pinpoint where the data structure is changing unexpectedly.

6. If the data structure is intentionally changing between iterations, you may need to add logic to handle different input formats in your source_tests task.

Remember, when working with Step Functions, it's crucial to ensure that the output of each state matches the expected input of the next state, especially when dealing with loops or multiple iterations.

Sources
Empty payload in the Pass step of Step Function Step Machine when triggered by an EventBridge event | AWS re:Post
Step Function Error Handling | AWS re:Post

The issue comes down to how Step Functions is handling your source_tests input after the first run.

Form the error: Cannot deserialize value of type java.util.ArrayList<java.lang.String> from Object value (token JsonToken.START_OBJECT)

Cannot deserialize value of type java.util.ArrayList<java.lang.String> from Object value (token JsonToken.START_OBJECT) …it looks like the ECS task is expecting a list of strings for the Command, but it’s receiving an object instead. That usually means your original array (the list of command arguments) got overwritten somewhere along the way, likely by the result of a previous step.

What’s likely happening: In your first run, source_tests is a clean list like: [ "./dbtWrapper.sh", "test", "--select", "source:mongo_test_event_source", "--target", "dev", "--vars", "{...}" ]

But after the ECS task finishes, if you're using something like "ResultPath": "$.source_tests", then Step Functions will replace source_tests with the ECS task result, which is a JSON object, and not valid for Command.

So in the second loop, it tries to use that object as the command, and ECS throws the deserialization error.

How to fix it: To avoid that conflict, you can:

1. Change the ResultPath in your ECS task step to store the result somewhere else, like $.source_tests_result

2. Or keep your command input inside a nested object like $.input.source_tests

For example: "ResultPath": "$.source_tests_result", "Command.$": "$.input.source_tests"

and update ypur json to

"input": { "source_tests": [ ... ] }

This keeps your original command array intact across multiple loops.

Hey, glad to hear the StartExecution permissions are working now, and great question about chaining executions and resetting the counter.

1. When the current Step Function ends immediately after triggering a new one, there’s a chance the new execution might not start before the old one exits. Step Functions' StartExecution is asynchronous, meaning the parent just triggers the new one and moves on.

If you're using a concurrency check at the start of the next execution, it's possible that the original execution hasn't ended yet when the new one checks.

In practice, this usually resolves itself quickly, but if strict ordering matters, you can add a short Wait state (e.g., 5–10 seconds) after the StartExecution to give the new run a buffer before doing the check.

2. If you're passing the whole input (Input.$: "$"), and iteration_count is already incremented in the current run, that value will carry over. To override it, just use Parameters to rebuild the input with the fields you want: "start_new_exec": { "Type": "Task", "Resource": "arn:aws:states:::states:startExecution", "Parameters": { "StateMachineArn.$": "$.step_function_arn", "Input": { "iteration_count": 0, "continue_loop_flag": "true", "concurrency": { "Payload": { "retryCount": 0, "waitingInstanceDetails": [], "otherStepFunctionArns.$": "$.concurrency.Payload.otherStepFunctionArns" } }, "source_tests.$": "$.source_tests", "placement_transform.$": "$.placement_transform", "placement_post_transformation_tests.$": "$.placement_post_transformation_tests" } }, "End": true }

Basically: rebuild the Input payload manually and just reset the fields you want, like iteration_count = 0.

Great to hear you've implemented all the core steps. When it comes to managing parallel Step Function executions (especially when triggered by a scheduler or chaining logic), you're right, it can cause race conditions or unexpected behavior if not controlled.

Here are a few practical options to manage and avoid unwanted parallel executions:

1. Use a Concurrency Guard via DynamoDB or S3 Set a "lock" at the beginning of the Step Function using a DynamoDB item or a marker file in S3. Your flow would:

Check if an execution is already running (based on timestamp or status flag) If yes, exit early or wait If no, write the lock and continue Clear the lock at the end or upon failure (with Catch block fallback) This works well with EventBridge triggers or scheduled runs.

2. Use a Custom EventBridge Rule with Input Filtering Instead of triggering your Step Function directly, you can send an event to EventBridge with metadata (e.g., timestamp, job ID), and have a Lambda in between that:

Validates whether a Step Function is already running (maybe by listing executions with DescribeExecutions)

If nothing is active → trigger Step Function

If one is running → skip or reschedule (e.g., via SQS DLQ or another EventBridge rule)

This gives you more centralized control and decouples the trigger logic.

3. Use Step Functions' Built-in Execution ID Control (if applicable). If you're triggering via StartExecution, you can provide a custom name for the execution. AWS will return an error if one with the same name is already running. You can catch that error and decide to skip or retry later.

Example: "ExecutionName": "my-daily-run-{{timestamp}}" Or use a static name like "daily-job" and enforce one-at-a-time runs.

4. Move the Scheduler Outside and Use Conditional Triggering If the scheduler is what's introducing duplicates, consider replacing it with:

An EventBridge rule that triggers based on a condition (e.g., CloudWatch metric, status check)

Or use a control Step Function that decides whether to launch the actual processing flow (this is sometimes called a "controller–worker" model)