AWS Batch: Running ECSProperties Job with AWS Stepfunction

Question

I have AWS Stepfunction that starts with a Lambda function to prepare the execution of an AWS Batch Job, of which the Job Definition specifies to use Fargate (ecsProperties Job). This stepfunction fails at the `submit-batch-job` step:

```
{
  "Comment": "AWS Step Functions for processing batch jobs and updating Athena",
  "StartAt": "Prepare Batch Job",
  "States": {
    "Prepare Batch Job": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:::function:prepare-batch-job",
      "Next": "Run Batch Job"
    },
    "Run Batch Job": {
      "Type": "Task",
      "Resource": "arn:aws:states:::batch:submitJob.sync",
      "Parameters": {
        "JobName.$": "$.jobName",
        "JobQueue.$": "$.jobQueue",
        "JobDefinition.$": "$.jobDefinition",
        "ArrayProperties": {
          "Size.$": "$.number_of_batches"
        },
        "Parameters": {
          "table_id.$": "$.table_id",
          "run_timestamp.$": "$.run_timestamp",
          "table_path_s3.$": "$.table_path_s3",
          "batches_s3_path.$": "$.batches_s3_path",
          "is_training_run.$": "$.is_training_run"
        }
      },
      "Next": "Prepare Athena Query"
    },
...
```

Upon execution, the `Run Batch Job` step fails with the following message:

`Container overrides should not be set for ecsProperties jobs. (Service: AWSBatch; Status Code: 400; Error Code: ClientException; Request ID: ffewfwe96-c869-4106-bc4d-3cfd6c7c34a0; Proxy: null)`

One very important thing to note is that, if I move the submit-job request to the first step (lambda) using the [boto3 api](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch/client/submit_job.html), the job gets submitted and starts running without issues. However, when I submit the job from the `Run Batch Job` step within the stepfunction, the aforementioned error appears.

This question has already been posted [here](https://repost.aws/questions/QUHzpyD5gGQ2ic4TJsJ-U3Hw/the-error-occurred-when-calling-aws-batch-ecsproperties-job-from-aws-step-functions), wherein the author notes that AWS Stepfunctions automatically adds the following to the definition, which appears to be the root of the error:

```
  "ContainerOverrides":{
    "Environment": [
      {
        "Name": "MANAGED_BY_AWS",
        "Value": "STARTED_BY_STEP_FUNCTIONS"
      }
    ]
  }
```

The answer provided in the aforementioned post however seems unclear to me as someone who has only started using AWS Batch a short while ago. If anyone would care to elaborate and assist, I would be very grateful.

I should state that the only reason I need to use the `Run Batch Job` step approach, is that I need my workflow to wait for the batch job to complete before attempting to insert the results as a new partition into an Athena results table. This is not feasible from within the Lambda function using boto3, as Lambdas timeout after 15 minutes, and the boto3 submit_job method does not wait for the execution to complete.

Thanks in advance.

re:Post Agent · Answer

The issue you're encountering is related to how AWS Step Functions interacts with AWS Batch, particularly for jobs using Fargate (ecsProperties jobs). Step Functions automatically adds container overrides to the job submission, which is not compatible with ecsProperties jobs.

To resolve this, you need to use a different approach when submitting Fargate-based AWS Batch jobs from Step Functions. Instead of using the `arn:aws:states:::batch:submitJob.sync` resource, you should use `arn:aws:states:::batch:submitJob.sync-2`.

Here's how you can modify your "Run Batch Job" state:

```json
"Run Batch Job": {
  "Type": "Task",
  "Resource": "arn:aws:states:::batch:submitJob.sync-2",
  "Parameters": {
    "JobName.The issue you're encountering is related to how AWS Step Functions interacts with AWS Batch, particularly for jobs using Fargate (ecsProperties jobs). Step Functions automatically adds container overrides to the job submission, which is not compatible with ecsProperties jobs.

To resolve this, you need to use a different approach when submitting Fargate-based AWS Batch jobs from Step Functions. Instead of using the `arn:aws:states:::batch:submitJob.sync` resource, you should use `arn:aws:states:::batch:submitJob.sync-2`.

Here's how you can modify your "Run Batch Job" state:

```json
"Run Batch Job": {
  "Type": "Task",
  "Resource": "arn:aws:states:::batch:submitJob.sync-2",
  "Parameters": {
    "JobName.$": "$.jobName",
    "JobQueue.$": "$.jobQueue",
    "JobDefinition.$": "$.jobDefinition",
    "ArrayProperties": {
      "Size.$": "$.number_of_batches"
    },
    "Parameters": {
      "table_id.$": "$.table_id",
      "run_timestamp.$": "$.run_timestamp",
      "table_path_s3.$": "$.table_path_s3",
      "batches_s3_path.$": "$.batches_s3_path",
      "is_training_run.$": "$.is_training_run"
    }
  },
  "Next": "Prepare Athena Query"
}
```

The `submitJob.sync-2` resource is designed to work with Fargate-based jobs and doesn't add the problematic container overrides. This should allow your Step Function to submit the Batch job successfully and wait for its completion before moving to the next state.

This approach addresses your need to wait for the batch job to complete before proceeding to insert results into the Athena table, without running into Lambda timeout issues.

Remember that when using `submitJob.sync-2`, Step Functions will wait for the job to complete before moving to the next state, which aligns with your workflow requirements.quot;: "$.jobName",
    "JobQueue.The issue you're encountering is related to how AWS Step Functions interacts with AWS Batch, particularly for jobs using Fargate (ecsProperties jobs). Step Functions automatically adds container overrides to the job submission, which is not compatible with ecsProperties jobs.

To resolve this, you need to use a different approach when submitting Fargate-based AWS Batch jobs from Step Functions. Instead of using the `arn:aws:states:::batch:submitJob.sync` resource, you should use `arn:aws:states:::batch:submitJob.sync-2`.

Here's how you can modify your "Run Batch Job" state:

```json
"Run Batch Job": {
  "Type": "Task",
  "Resource": "arn:aws:states:::batch:submitJob.sync-2",
  "Parameters": {
    "JobName.$": "$.jobName",
    "JobQueue.$": "$.jobQueue",
    "JobDefinition.$": "$.jobDefinition",
    "ArrayProperties": {
      "Size.$": "$.number_of_batches"
    },
    "Parameters": {
      "table_id.$": "$.table_id",
      "run_timestamp.$": "$.run_timestamp",
      "table_path_s3.$": "$.table_path_s3",
      "batches_s3_path.$": "$.batches_s3_path",
      "is_training_run.$": "$.is_training_run"
    }
  },
  "Next": "Prepare Athena Query"
}
```

The `submitJob.sync-2` resource is designed to work with Fargate-based jobs and doesn't add the problematic container overrides. This should allow your Step Function to submit the Batch job successfully and wait for its completion before moving to the next state.

This approach addresses your need to wait for the batch job to complete before proceeding to insert results into the Athena table, without running into Lambda timeout issues.

Remember that when using `submitJob.sync-2`, Step Functions will wait for the job to complete before moving to the next state, which aligns with your workflow requirements.quot;: "$.jobQueue",
    "JobDefinition.The issue you're encountering is related to how AWS Step Functions interacts with AWS Batch, particularly for jobs using Fargate (ecsProperties jobs). Step Functions automatically adds container overrides to the job submission, which is not compatible with ecsProperties jobs.

To resolve this, you need to use a different approach when submitting Fargate-based AWS Batch jobs from Step Functions. Instead of using the `arn:aws:states:::batch:submitJob.sync` resource, you should use `arn:aws:states:::batch:submitJob.sync-2`.

Here's how you can modify your "Run Batch Job" state:

```json
"Run Batch Job": {
  "Type": "Task",
  "Resource": "arn:aws:states:::batch:submitJob.sync-2",
  "Parameters": {
    "JobName.$": "$.jobName",
    "JobQueue.$": "$.jobQueue",
    "JobDefinition.$": "$.jobDefinition",
    "ArrayProperties": {
      "Size.$": "$.number_of_batches"
    },
    "Parameters": {
      "table_id.$": "$.table_id",
      "run_timestamp.$": "$.run_timestamp",
      "table_path_s3.$": "$.table_path_s3",
      "batches_s3_path.$": "$.batches_s3_path",
      "is_training_run.$": "$.is_training_run"
    }
  },
  "Next": "Prepare Athena Query"
}
```

The `submitJob.sync-2` resource is designed to work with Fargate-based jobs and doesn't add the problematic container overrides. This should allow your Step Function to submit the Batch job successfully and wait for its completion before moving to the next state.

This approach addresses your need to wait for the batch job to complete before proceeding to insert results into the Athena table, without running into Lambda timeout issues.

Remember that when using `submitJob.sync-2`, Step Functions will wait for the job to complete before moving to the next state, which aligns with your workflow requirements.quot;: "$.jobDefinition",
    "ArrayProperties": {
      "Size.The issue you're encountering is related to how AWS Step Functions interacts with AWS Batch, particularly for jobs using Fargate (ecsProperties jobs). Step Functions automatically adds container overrides to the job submission, which is not compatible with ecsProperties jobs.

To resolve this, you need to use a different approach when submitting Fargate-based AWS Batch jobs from Step Functions. Instead of using the `arn:aws:states:::batch:submitJob.sync` resource, you should use `arn:aws:states:::batch:submitJob.sync-2`.

Here's how you can modify your "Run Batch Job" state:

```json
"Run Batch Job": {
  "Type": "Task",
  "Resource": "arn:aws:states:::batch:submitJob.sync-2",
  "Parameters": {
    "JobName.$": "$.jobName",
    "JobQueue.$": "$.jobQueue",
    "JobDefinition.$": "$.jobDefinition",
    "ArrayProperties": {
      "Size.$": "$.number_of_batches"
    },
    "Parameters": {
      "table_id.$": "$.table_id",
      "run_timestamp.$": "$.run_timestamp",
      "table_path_s3.$": "$.table_path_s3",
      "batches_s3_path.$": "$.batches_s3_path",
      "is_training_run.$": "$.is_training_run"
    }
  },
  "Next": "Prepare Athena Query"
}
```

The `submitJob.sync-2` resource is designed to work with Fargate-based jobs and doesn't add the problematic container overrides. This should allow your Step Function to submit the Batch job successfully and wait for its completion before moving to the next state.

This approach addresses your need to wait for the batch job to complete before proceeding to insert results into the Athena table, without running into Lambda timeout issues.

Remember that when using `submitJob.sync-2`, Step Functions will wait for the job to complete before moving to the next state, which aligns with your workflow requirements.quot;: "$.number_of_batches"
    },
    "Parameters": {
      "table_id.The issue you're encountering is related to how AWS Step Functions interacts with AWS Batch, particularly for jobs using Fargate (ecsProperties jobs). Step Functions automatically adds container overrides to the job submission, which is not compatible with ecsProperties jobs.

To resolve this, you need to use a different approach when submitting Fargate-based AWS Batch jobs from Step Functions. Instead of using the `arn:aws:states:::batch:submitJob.sync` resource, you should use `arn:aws:states:::batch:submitJob.sync-2`.

Here's how you can modify your "Run Batch Job" state:

```json
"Run Batch Job": {
  "Type": "Task",
  "Resource": "arn:aws:states:::batch:submitJob.sync-2",
  "Parameters": {
    "JobName.$": "$.jobName",
    "JobQueue.$": "$.jobQueue",
    "JobDefinition.$": "$.jobDefinition",
    "ArrayProperties": {
      "Size.$": "$.number_of_batches"
    },
    "Parameters": {
      "table_id.$": "$.table_id",
      "run_timestamp.$": "$.run_timestamp",
      "table_path_s3.$": "$.table_path_s3",
      "batches_s3_path.$": "$.batches_s3_path",
      "is_training_run.$": "$.is_training_run"
    }
  },
  "Next": "Prepare Athena Query"
}
```

The `submitJob.sync-2` resource is designed to work with Fargate-based jobs and doesn't add the problematic container overrides. This should allow your Step Function to submit the Batch job successfully and wait for its completion before moving to the next state.

This approach addresses your need to wait for the batch job to complete before proceeding to insert results into the Athena table, without running into Lambda timeout issues.

Remember that when using `submitJob.sync-2`, Step Functions will wait for the job to complete before moving to the next state, which aligns with your workflow requirements.quot;: "$.table_id",
      "run_timestamp.The issue you're encountering is related to how AWS Step Functions interacts with AWS Batch, particularly for jobs using Fargate (ecsProperties jobs). Step Functions automatically adds container overrides to the job submission, which is not compatible with ecsProperties jobs.

To resolve this, you need to use a different approach when submitting Fargate-based AWS Batch jobs from Step Functions. Instead of using the `arn:aws:states:::batch:submitJob.sync` resource, you should use `arn:aws:states:::batch:submitJob.sync-2`.

Here's how you can modify your "Run Batch Job" state:

```json
"Run Batch Job": {
  "Type": "Task",
  "Resource": "arn:aws:states:::batch:submitJob.sync-2",
  "Parameters": {
    "JobName.$": "$.jobName",
    "JobQueue.$": "$.jobQueue",
    "JobDefinition.$": "$.jobDefinition",
    "ArrayProperties": {
      "Size.$": "$.number_of_batches"
    },
    "Parameters": {
      "table_id.$": "$.table_id",
      "run_timestamp.$": "$.run_timestamp",
      "table_path_s3.$": "$.table_path_s3",
      "batches_s3_path.$": "$.batches_s3_path",
      "is_training_run.$": "$.is_training_run"
    }
  },
  "Next": "Prepare Athena Query"
}
```

The `submitJob.sync-2` resource is designed to work with Fargate-based jobs and doesn't add the problematic container overrides. This should allow your Step Function to submit the Batch job successfully and wait for its completion before moving to the next state.

This approach addresses your need to wait for the batch job to complete before proceeding to insert results into the Athena table, without running into Lambda timeout issues.

Remember that when using `submitJob.sync-2`, Step Functions will wait for the job to complete before moving to the next state, which aligns with your workflow requirements.quot;: "$.run_timestamp",
      "table_path_s3.The issue you're encountering is related to how AWS Step Functions interacts with AWS Batch, particularly for jobs using Fargate (ecsProperties jobs). Step Functions automatically adds container overrides to the job submission, which is not compatible with ecsProperties jobs.

To resolve this, you need to use a different approach when submitting Fargate-based AWS Batch jobs from Step Functions. Instead of using the `arn:aws:states:::batch:submitJob.sync` resource, you should use `arn:aws:states:::batch:submitJob.sync-2`.

Here's how you can modify your "Run Batch Job" state:

```json
"Run Batch Job": {
  "Type": "Task",
  "Resource": "arn:aws:states:::batch:submitJob.sync-2",
  "Parameters": {
    "JobName.$": "$.jobName",
    "JobQueue.$": "$.jobQueue",
    "JobDefinition.$": "$.jobDefinition",
    "ArrayProperties": {
      "Size.$": "$.number_of_batches"
    },
    "Parameters": {
      "table_id.$": "$.table_id",
      "run_timestamp.$": "$.run_timestamp",
      "table_path_s3.$": "$.table_path_s3",
      "batches_s3_path.$": "$.batches_s3_path",
      "is_training_run.$": "$.is_training_run"
    }
  },
  "Next": "Prepare Athena Query"
}
```

The `submitJob.sync-2` resource is designed to work with Fargate-based jobs and doesn't add the problematic container overrides. This should allow your Step Function to submit the Batch job successfully and wait for its completion before moving to the next state.

This approach addresses your need to wait for the batch job to complete before proceeding to insert results into the Athena table, without running into Lambda timeout issues.

Remember that when using `submitJob.sync-2`, Step Functions will wait for the job to complete before moving to the next state, which aligns with your workflow requirements.quot;: "$.table_path_s3",
      "batches_s3_path.The issue you're encountering is related to how AWS Step Functions interacts with AWS Batch, particularly for jobs using Fargate (ecsProperties jobs). Step Functions automatically adds container overrides to the job submission, which is not compatible with ecsProperties jobs.

To resolve this, you need to use a different approach when submitting Fargate-based AWS Batch jobs from Step Functions. Instead of using the `arn:aws:states:::batch:submitJob.sync` resource, you should use `arn:aws:states:::batch:submitJob.sync-2`.

Here's how you can modify your "Run Batch Job" state:

```json
"Run Batch Job": {
  "Type": "Task",
  "Resource": "arn:aws:states:::batch:submitJob.sync-2",
  "Parameters": {
    "JobName.$": "$.jobName",
    "JobQueue.$": "$.jobQueue",
    "JobDefinition.$": "$.jobDefinition",
    "ArrayProperties": {
      "Size.$": "$.number_of_batches"
    },
    "Parameters": {
      "table_id.$": "$.table_id",
      "run_timestamp.$": "$.run_timestamp",
      "table_path_s3.$": "$.table_path_s3",
      "batches_s3_path.$": "$.batches_s3_path",
      "is_training_run.$": "$.is_training_run"
    }
  },
  "Next": "Prepare Athena Query"
}
```

The `submitJob.sync-2` resource is designed to work with Fargate-based jobs and doesn't add the problematic container overrides. This should allow your Step Function to submit the Batch job successfully and wait for its completion before moving to the next state.

This approach addresses your need to wait for the batch job to complete before proceeding to insert results into the Athena table, without running into Lambda timeout issues.

Remember that when using `submitJob.sync-2`, Step Functions will wait for the job to complete before moving to the next state, which aligns with your workflow requirements.quot;: "$.batches_s3_path",
      "is_training_run.The issue you're encountering is related to how AWS Step Functions interacts with AWS Batch, particularly for jobs using Fargate (ecsProperties jobs). Step Functions automatically adds container overrides to the job submission, which is not compatible with ecsProperties jobs.

To resolve this, you need to use a different approach when submitting Fargate-based AWS Batch jobs from Step Functions. Instead of using the `arn:aws:states:::batch:submitJob.sync` resource, you should use `arn:aws:states:::batch:submitJob.sync-2`.

Here's how you can modify your "Run Batch Job" state:

```json
"Run Batch Job": {
  "Type": "Task",
  "Resource": "arn:aws:states:::batch:submitJob.sync-2",
  "Parameters": {
    "JobName.$": "$.jobName",
    "JobQueue.$": "$.jobQueue",
    "JobDefinition.$": "$.jobDefinition",
    "ArrayProperties": {
      "Size.$": "$.number_of_batches"
    },
    "Parameters": {
      "table_id.$": "$.table_id",
      "run_timestamp.$": "$.run_timestamp",
      "table_path_s3.$": "$.table_path_s3",
      "batches_s3_path.$": "$.batches_s3_path",
      "is_training_run.$": "$.is_training_run"
    }
  },
  "Next": "Prepare Athena Query"
}
```

The `submitJob.sync-2` resource is designed to work with Fargate-based jobs and doesn't add the problematic container overrides. This should allow your Step Function to submit the Batch job successfully and wait for its completion before moving to the next state.

This approach addresses your need to wait for the batch job to complete before proceeding to insert results into the Athena table, without running into Lambda timeout issues.

Remember that when using `submitJob.sync-2`, Step Functions will wait for the job to complete before moving to the next state, which aligns with your workflow requirements.quot;: "$.is_training_run"
    }
  },
  "Next": "Prepare Athena Query"
}
```

The `submitJob.sync-2` resource is designed to work with Fargate-based jobs and doesn't add the problematic container overrides. This should allow your Step Function to submit the Batch job successfully and wait for its completion before moving to the next state.

This approach addresses your need to wait for the batch job to complete before proceeding to insert results into the Athena table, without running into Lambda timeout issues.

Remember that when using `submitJob.sync-2`, Step Functions will wait for the job to complete before moving to the next state, which aligns with your workflow requirements.  
**Sources**  
[The error occurred when calling AWS Batch (ECSProperties job) from AWS Step Functions. | AWS re:Post](https://repost.aws/questions/QUHzpyD5gGQ2ic4TJsJ-U3Hw/the-error-occurred-when-calling-aws-batch-ecsproperties-job-from-aws-step-functions)  
[Discover service integration patterns in Step Functions - AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html)

AWS Batch: Running ECSProperties Job with AWS Stepfunction

Add your answer

Relevant content