Questions tagged with AWS Batch

How do I create a role for AWS Batch jobs on EC2 using the CLI? The role needs to have read-write permissions to an S3 bucket, e.g. s3://mybucket234. The role will be needed for the "aws batch create-compute-environment" command.

Hello, I am using Kinesis Firehose and saving the raw streamed data into json files in S3. I am using the standard firehose partition <stream_name>/YYYY/MM/DD/HH. For the data that is really urgent as soon as the file is saved into s3 a lambada function is triggered to process the data in the file. Other data doesn't have the same urgency so we can do batches every 5 or 10 minutes. My question is related to the data that can be processed in batches. I don't know what processing strategy or methodology I should implement so every time the batch runs it will only process the json files that have not been processed before. For example it is 2022-01-28 14:15:00 . We have 2 files in the same partition. My process runs and loads those 2 files. Then at 2022-01-28 14:25:00 the process runs again and there are 3 files in the partition. The previous batch already processed 2 of those files so the new batch should only process one file. How can I know which files were already processed so I don't read them again on my next batch? I was planning to use Airflow to schedule some spark jobs to do the batch processing. What tool or technology would you recommend for doing this kind of batches?

I've created a docker image, pushed it into a private ECR Repository, and configured an AWS Batch cluster/queue/job definition. When I submit a job, it immediately goes to the *STARTING* state, and then fails with ``` ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post https://api.ecr.us-west-2.amazonaws.com/: dial tcp 54.240.255.116:443: i/o timeout ``` This seems to be a problem with the container image not being pulled. My cluster has the following specs: * Fargate provision model * Default VPC * Default security group (allows all outbound traffic, but only inbound from the default SG) * Default subnets (4 subnets with a route to an internet gateway and a single ACL rule allowing all traffic) The job definition has an execution role with the managed policy `AmazonECSTaskExecutionRolePolicy` and has the "Public IP" option **disabled**. The network configuration seems to be enough to pull images from the internet, but I'm still getting the timeout error. Also, the IAM Role seems to have the relevant policies to authenticate with my private ECR. Can someone help me debug this?

When using AWS Batch is there a way to limit the maximum number of EC2 instances that AWS Batch will have running at one time? Ideally, this would apply across all compute instances. Does AWS Batch enable this or are there other AWS services that can be combined with AWS Batch to allow us to achieve instance based limits? An example would be say I am running AWS Batch for some workload. But I want AWS Batch to at most have 10 EC2 instances running. Is there somewhere that I can configure AWS Batch such that it will respect that limit on EC2 instances?

My system connects to share brokers using REST APIs during trading hours. Each thread takes care of a user account. Based on the number of my users several threads have to run simulationsly. Each thread takes care of a user account with a share broker. Can I use AWS Batch service for this usecase? Or is it better to use several Amazon ec2s by calculating memory and processor requirements?

I'm trying to setup an aws batch job to run daily. Through the AWS batch GUI I've created the job definition, job queue & compute environment. The job can successfully run when, via batch, a new job is manually configured & submitted. I'm now trying to use an Event Bridge rule to automate the daily job execution but running into an issue. The trigger runs but returns the following error (from CloudTrail): ```json "responseElements": { "message": "shareIdentifier must be specified.", "__type": "ClientException" }, ``` When manually executing the batch job there's an option to enter a `share identifier` but when configuring the Event Bridge rule this option doesn't exist. When defining the Event Bridge rule, the following configuration has been used: * Rule Type: Schedule * Target: * Target Type: AWS Service * Select a target: Batch Job queue * The job queue, definition & name are entered Further to this, I've tried to configure target inputs (`Constant (JSON text) `) & tried the following approaches: ### First Attempt ```json { "ContainerOverrides": { "shareIdentifier": "*" } } ``` ### Second Attempt ```json { "schedulingPriorityOverride": 1, "shareIdentifier": "*" } ``` But none of these have worked. In fact, when examining Cloud Trail neither attempt appears in `requestParameters`. Any suggestions or working examples for how to solve it?

I would like to sell a software as a service on AWS marketplace. The server side of my software (there is also a client side installed on customer machine) is supposed to run using aws batch service on my account because I do not want to expose the code to customers. The idea was to create a product based on a small docker container dealing with user registration and other administrative tasks. The batch jobs should be lunched using lambda functions connected to an API gateway. Is it possible to bill a marketplace customer for used resources calling RegisterUsage API from batch jobs in the seller account ?

``` { "Comment": "A description of my state machine", "StartAt": "Batch SubmitJob", "States": { "Batch SubmitJob": { "Type": "Task", "Resource": "arn:aws:states:::batch:submitJob.sync", "Parameters": { "JobDefinition": "arn:aws:batch:us-east-1:XXXXXXXXXXX:job-definition/clientcopyjobdef:1", "JobQueue": "arn:aws:batch:us-east-1:XXXXXXXXXX:job-queue/copyclientjq", "ContainerOverrides": { "Command.$": [ "dotnet", "CopyClientJob.dll", "$.input" ] }, "JobName.$": "$.input" }, "End": true } } } ``` I was trying to create this state machine, it was working fine if I am passing command to JobDefinition directly, but here I am truing to override the command and want to pass command parameter from state input, so trying to pass like above code. For `"JobName.$": "$.input"` it is working but for ``` "Command.$": [ "dotnet", "CopyClientJob.dll", "$.input" ] ``` It is not working, the command is passing as it is to aws batch without trasforming the parameter, can someone help on this. Thanks

Hi, I keep getting this error: AttributeError: module 'sagemaker' has no attribute 'create_transform_job', when using a batch transform example that AWS graciously had in the notebook instances. Code: ***Also, I updated Sagemaker to the newest package and its still not working. ``` %%time import time from time import gmtime, strftime batch_job_name = "Batch-Transform-" + strftime("%Y-%m-%d-%H-%M-%S", gmtime()) input_location = "s3://{}/{}/batch/{}".format( bucket, prefix, batch_file ) # use input data without ID column output_location = "s3://{}/{}/output/{}".format(bucket, prefix, batch_job_name) request = { "TransformJobName": batch_job_name, "ModelName": 'xgboost-parquet-example-training-2022-03-28-16-02-31-model', "TransformOutput": { "S3OutputPath": output_location, "Accept": "text/csv", "AssembleWith": "Line", }, "TransformInput": { "DataSource": {"S3DataSource": {"S3DataType": "S3Prefix", "S3Uri": input_location}}, "ContentType": "text/csv", "SplitType": "Line", "CompressionType": "None", }, "TransformResources": {"InstanceType": "ml.m4.xlarge", "InstanceCount": 1}, } sagemaker.create_transform_job(**request) print("Created Transform job with name: ", batch_job_name) # Wait until the job finishes try: sagemaker.get_waiter("transform_job_completed_or_stopped").wait(TransformJobName=batch_job_name) finally: response = sagemaker.describe_transform_job(TransformJobName=batch_job_name) status = response["TransformJobStatus"] print("Transform job ended with status: " + status) if status == "Failed": message = response["FailureReason"] print("Transform failed with the following error: {}".format(message)) raise Exception("Transform job failed") ``` Everything else is working well. I've had no luck with this on anyother forum.

I have an AWS Batch setup, with: - a Compute Environment of type MANAGED configured to have Min vCPUs = 0 - Max vCPUs = 512, and allocation strategy SPOT_CAPACITY_OPTIMIZED (max % on-demand price set to 100%) - a Job Definition targetting 16 vCPUs and Memory 16,384. I often trigger hundreds of jobs at the same time, none of which apply any overwrites to this JD. What I don't understand is why I hardly ever have more than about 26 ECS tasks running at the same time. I'm looking at it now and only seeing 13 m4.10xlarge instances running (40 vCPUs each, so presumably 2 tasks per instance == 26 tasks), whilst I still have more than 100 tasks in RUNNABLE state... Right now the Desired vCPUs is showing "480" (but not sure what that means). I have another JD asking for 64 vCPUs and 65536 memory (for bigger tasks) and tried testing another "spot" CE with Max vCPUs 4096, thinking the larger number would get me more instances more quickly. Just the opposite in fact. I had just 6 R4.x16large instances running at the same time, and the whole thing took many hours How can I configure things to use as many instances in parallel to clear the queue as quickly as possible? (In case it's relevant, I'm using eu-west-1)

I am trying to automate Fargate AWS Batch jobs by means of AWS Cloudwatch Events. So far, so good. I am trying to run the same job definition with different configurations. I am able to set the batch job as a cloudwatch event target. I have learned how to use the Constant (JSON text) configuration to set a parameter of the job. Thus, I can set the name parameter successfully and the job runs. However, I am not able to also set the memory and cpu settings in the Cloudwatch event. I would like to use a larger machine for a a bigger port such as Singapore, without changing the job definition. After all, at the moment it still uses the default vpcu and memory settings of the job definition. ``` { "Parameters": {"name":"wilhelmshaven"}, "ContainerOverrides": { "Command": ["upload_to_day.py", "-port_name","Ref::name"], "resourceRequirements": [ {"type": "MEMORY", "value": "4096"}, {"type": "VCPU", "value": "2"} ] } } ``` Does any one know how to set the Constant (JSON text) configuration or input transformer correctly? Edit: If I try the same thing using the AWS CLI, I can achieve what I would like to do. ``` aws batch submit-job \ --job-name "run-wilhelmshaven" \ --job-queue "arn:aws:batch:eu-central-1:123666072061:job-queue/upload-raw-to-day-vtexplorer" \ --job-definition "arn:aws:batch:eu-central-1:123666072061:job-definition/upload-to-day:2" \ --container-overrides '{"command": ["upload_to_day.py", "-port_name","wilhelmshaven"], "resourceRequirements": [{"value": "2", "type": "VCPU"}, {"value": "4096", "type": "MEMORY"}]}' ```

Is there a recommended Docker image / Dockerfile starting point for running R scripts in AWS Batch? I have looked at a few here, and the starting image is could be any of the following: `rocker`, `python:3.6`, `ubuntu:16.04` or `r-base`: - (FROM rocker) https://github.com/aws-samples/aws-fargate-with-rstudio-open-source/blob/main/docimage-rstudio/Dockerfile - (FROM rocker) https://github.com/aws-samples/aws-fargate-with-rstudio-open-source/blob/main/docimage-shiny/Dockerfile - (FROM python:3.6) https://github.com/aws-samples/sagemaker-studio-custom-image-samples/blob/main/examples/r-image/Dockerfile - (FROM ubuntu:16.04) https://github.com/aws/amazon-sagemaker-examples/blob/main/hyperparameter_tuning/r_bring_your_own/Dockerfile - (FROM r-base) https://github.com/aws-samples/reinvent2020-aim404-productionize-r-using-amazon-sagemaker/blob/main/Dockerfile What are the key points to consider in choosing a good starting point for a Dockerfile for R jobs? For this particular workload, my main R packages are `data.table` and the tidyverse packages (no need for RStudio / Shiny server).

Can a regular (non-machine learning) Python job submitted via AWS Batch make use of SageMaker [FastFile mode](https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-sagemaker-fast-file-mode/) to stream data from S3 to the container? Or is there any equivalent of SageMaker FastFile mode in other Python libraries? The key point is about whether it's possible to avoid the need to copy data from S3 to the EBS or instance store volume before processing.

For a job, I run puppeteer in the headful mode in AWS Batch. I found a docker file that uses the image `node:16.5-slim`, then downloads Chrome, and then uses Xvfb. If I run it on an EC2 instance with GPU, Chrome still uses the integrated video card (llvmpipe) not taking advantage of the powerful Nvidia GPU. I found online a couple of attempts, one I heard they had to switch to Windows (not supported by AWS Batch). They also used an AMI only available on EC2: https://stackoverflow.com/questions/67813309/run-headful-chromium-with-nvidia-gpu-and-xvfb-on-ec2-instance-with-prime And the other solution also doesn't seem to be suitable because of its complexity to set up: https://boygiandi.medium.com/ch%E1%BA%A1y-chrome-v%E1%BB%9Bi-gpu-tr%C3%AAn-instance-ec2-aws-69369d73235a I wonder if there is a docker image I could use as a starting point where the Nvidia GPU is already set up and Chrome could take advantage of it with only a few extra configurations. Thanks!

My aws elemental media convert is showing Network failure message failure code networking error time 2022-03-16T14 Retryable true Someone please help me

I am currently trying to transcode my set of videos through amazon batch trancode but am having problems.My cloudwatch log output this error. /var/runtime/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'tgzfwmguc.mediaconvert.eu-west-1.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings /var/runtime/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'tgzfwmguc.mediaconvert.eu-west-1.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings. My media convert job is showing this error. Error message Unable to open input file [s3://redacted/redacted.mp4]: [Failed probe/open: [Can't read input stream: [Failed to read data: HeadObject failed]]] Error code 1404

How do I update the docker image for a Batch Job Definition using CLI or API? It looks like the APIs for `RegisterJobDefinition` are "create only". You can't update a Job Definition from what I can tell from the documentation, so you can't change the reference to the Docker Image. The JobDefinition really wants to be defined in the CDK Constructs area (or CFT) because it ties in a bunch of stuff I already have in the CDK such as databases and EFS and Secrets. That is fine, as that all that stuff is fairly static. But Docker images are meant to change all the time, and quickly as my devs iterate code. I really don't want to specify the final Docker Image at creation time in CDK or CFT, but it looks like that's the only place to do it. I do not want to re-deploy a CDK/CFT instance just to change some code in a Docker container, that would be a slow and bad practice. Note: a[ similar was asked on the old forum](https://forums.aws.amazon.com/thread.jspa?threadID=257528&tstart=0), but didn't really get an answer. "use :latest" isn't always the best answer for Docker version management. My devs need to be able to iterate quickly and not walk over each other. I would like my devs to be able to change to a new Docker image and then test a batch. How can they do this quickly and easily? Note: Here's a[ blog post ](https://stevelasker.blog/2018/03/01/docker-tagging-best-practices-for-tagging-and-versioning-docker-images/)on `stable` tagging versus `unique` tagging. For deployments they recommend unique. Which isn't supported with BATCH, AFAICT.

For processing genomics datasets in containers launched from AWS Batch, is it possible to stream the data directly from S3, using an approach similar to SageMaker [pipe mode](https://aws.amazon.com/blogs/machine-learning/using-pipe-input-mode-for-amazon-sagemaker-algorithms/) (or [fast file mode](https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-sagemaker-fast-file-mode/))? Also, are there any hardware limitations that I should be aware of? For example, in [this article](https://aws.amazon.com/premiumsupport/knowledge-center/s3-maximum-transfer-speed-ec2/), the maximum transfer speed between EC2 and S3 is 100 Gbps (VPC endpoints and public IPs in the same region). In a 2018 article, the bandwidth between EC2 and S3 was [stated as 25 Gbps](https://aws.amazon.com/blogs/aws/the-floodgates-are-open-increased-network-bandwidth-for-ec2-instances/)

Hi, I'm trying to use an EFS volume in a Batch job running on AWS Fargate. The support for EFS volumes was added in Fargate version 1.4.0 (see [here](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/platform-linux-fargate.html)). In the linked article, it's also stated that using `LATEST` is equivalent to specifying version number `1.4.0`. However, when I try to specify the `LATEST` as a Fargate platform version in the Batch job definition where I also specify EFS volumes, I get this error: ``` Failed to register job definition Error: Error executing request, Exception : EFS is supported for Fargate platform version 1.4.0., RequestId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ``` Specifying an explicit version number `1.4.0` works without problems. If this is a bug, how to report the issue?

Hi, since AWS [announced](https://aws.amazon.com/about-aws/whats-new/2021/11/aws-fargate-amazon-ecs-aws-graviton2-processors/) Fargate support for ARM64 architecture I'm curious if there is any way for Batch jobs running on Fargate to use ARM64. For ECS tasks, we can specify this in `runtimePlatform` in the [task definition](https://docs.aws.amazon.com/AmazonECS/latest/userguide/ecs-arm64.html). I was looking into the [job definition parameters](https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html) for the same but there doesn't seem to be an option to specify ARM64. I also tried the brute force way of building the Docker image for ARM64 and running the job but it failed with exec format error. I couldn't find any mention for this topic nor the plans for adding the support (container roadmap). Thanks

Hello Everyone, I have setup AWS Batch Job with AWS RDS Postgresql as the backend. Earlier the jobs I submitted failed due to permission issues. But is fixed now. Earlier I was able to see the Logs and fix the issues. Now the jobs status shows as "SUCCEEDED". but the time it ran is zero and no logs are generated. Can someone guide me with this issue? Thanks in Advance Viddya

Whenever I try to update tags on an AWS::Batch::JobDefinition, the template deployment fails with "Internal Error". Without changing tags, no problem, with tags, internal error, whatever the tag name or value are. I'm using the latest AWS CLI Here is such a template: ``` MyJobDefinition: Type: AWS::Batch::JobDefinition Properties: Type: "container" Parameters: {} RetryStrategy: Attempts: 2 PropagateTags: true Tags: Version: "3.1.1" ContainerProperties: Command: - "python3" - "worker.py" JobRoleArn: Fn::ImportValue: !Sub "${IAMStackName}-EcsTaskExecutionRole" Image: Fn::Join: - ":" - - Fn::ImportValue: !Sub "${BaseStackName}-WorkerRepositoryUri" - !Ref DockerImageTag ResourceRequirements: - Type: VCPU Value: 16 - Type: MEMORY Value: 16384 ``` Steps to reproduce: - 1. Take the example CFN template https://s3-us-east-2.amazonaws.com/cloudformation-templates-us-east-2/Managed_EC2_Batch_Environment.template - 2. Add a Tag in the JobDefinition (eg. below) - 3. Deploy. All works - 4. Change the JobDefinition Tag in the template - 5. Deploy -> Internal Fail ``` "JobDefinition" : { "Type" : "AWS::Batch::JobDefinition", "Properties" : { "Type" : "container", "ContainerProperties" : { "Image" : { "Fn::Join": [ "", [ "137112412989.dkr.ecr.", { "Ref" : "AWS::Region" }, ".amazonaws.com/amazonlinux:latest" ]] }, "Vcpus" : 2, "Memory" : 2000, "Command" : [ "echo", "Hello world" ] }, "RetryStrategy" : { "Attempts" : 1 }, "Tags": { "svat-version": "3.1.1" } } }, ```

When creating an AWS Batch stack with CloudFormation, I'd like to be able to set tags to all resources, to track costs related to the stack. I noticed that it's possible to set a stack-level tag with `aws cloudformation deploy ... --tags tagName=tagValue`, and that the tag is added to all resources that support it. That's great, but there seems to be one crucial one missing: the EC2 tags in the ComputeEnvironment configuration. It seems to me that it's a crucial one when it comes to costs derived from the use of an AWS Batch stack. Anyone has any idea how to do something like that, without going back to having to manually add tags within the template on all resources?

We have an AWS Batch compute environment that is set up to use EC2 On-demand instances. Whenever we run a large number of tasks, Batch scales our group of instances up to the number of CPUs we require, but it never ends up scaling them back down, even when we have 0 Batch tasks running. Looking at the autoscaling group that Batch manages, I see it has scale-in protection enabled. This means it doesn't terminate EC2 instances when the desired capacity is decreased. So - looking at the autoscaling group's logs, it always scales up correctly, but scaling down fails because of the scale-in protection. For example: ``` Status: CANCELLED Error: Could not scale to desired capacity because all remaining instances are protected from scale-in. Cause: At 2022-02-02T13:11:53Z a user request update of AutoScalingGroup constraints to min: 0, max: 18, desired: 18 changing the desired capacity from 96 to provide the desired capacity of 18. At 2022-02-02T13:12:05Z group reached equilibrium. ``` My guess is this is why it isn't scaling down. But since this auto scaling group is managed by Batch and Batch should scale instances down relatively quickly, it doesn't make sense to me. This is costing us a lot in unused EC2 time, so if anyone can offer any insights, that'd be great.

Hi, I've a project to finish. My customer wants my docker containers's ip addresses has to change with a click. This is my first time in cloud. Is this possible, a dockerized running app's ip address can be changed. It can destroy and redeploy, but is this possible with a java based automatization. What should i study if it is possible. My app is spring boot micro service.

1)I have 150 ETL jobs need to be moved from on-premise to AWS cloud. What is the best way to schedule them in AWS? AWS batch or SWF or STEP function or any other service? Which tool/service has all the capability of a full time job scheduler? 2) As per AWS Batch documentation, it says that it supports all jobs that are Docker container. Does it mean that AWS batch is only meant for Docker container jobs? In my case all jobs are either bash/ksh scripts and normal ETL jobs that read from files from EC2 and load data into RDS DB.

I am trying to add a temporary volume to my Batch process for a machine learning application. I have created a Launch Template with a 104G drive and specified this, along with the appropriate amazon ML docker image in the compute environment. Instance types: optimal p3.2xlarge Ec2 configuration: ECS_AL2_NVIDIA The instance launches well enough, and the launch template seems to be respected, however I do not know how to use the additional drive. ls -al /dev ``` drwxr-xr-x 5 root root 380 Jan 23 03:00 . drwxr-xr-x 1 root root 101 Jan 23 03:00 .. lrwxrwxrwx 1 root root 11 Jan 23 03:00 core -> /proc/kcore lrwxrwxrwx 1 root root 13 Jan 23 03:00 fd -> /proc/self/fd crw-rw-rw- 1 root root 1, 7 Jan 23 03:00 full drwxrwxrwt 2 root root 40 Jan 23 03:00 mqueue crw-rw-rw- 1 root root 1, 3 Jan 23 03:00 null crw-rw-rw- 1 root root 195, 0 Jan 23 02:58 nvidia0 crw-rw-rw- 1 root root 195, 255 Jan 23 02:58 nvidiactl lrwxrwxrwx 1 root root 8 Jan 23 03:00 ptmx -> pts/ptmx drwxr-xr-x 2 root root 0 Jan 23 03:00 pts crw-rw-rw- 1 root root 1, 8 Jan 23 03:00 random drwxrwxrwt 2 root root 40 Jan 23 03:00 shm lrwxrwxrwx 1 root root 15 Jan 23 03:00 stderr -> /proc/self/fd/2 lrwxrwxrwx 1 root root 15 Jan 23 03:00 stdin -> /proc/self/fd/0 lrwxrwxrwx 1 root root 15 Jan 23 03:00 stdout -> /proc/self/fd/1 crw-rw-rw- 1 root root 5, 0 Jan 23 03:00 tty crw-rw-rw- 1 root root 1, 9 Jan 23 03:00 urandom crw-rw-rw- 1 root root 1, 5 Jan 23 03:00 zero ``` lsblk ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 30G 0 disk └─xvda1 202:1 0 30G 0 part /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.470.57.02 xvdb 202:16 0 104G 0 disk ``` The 104G disk is showing up as "xvdb" but there is no associated device file so I do not know how to use any of the mounting or formatting tools. I have tried changing the names and dive types in the launch template, but get exactly the same problem. I have tried adding random names and sources to the job definition "Volumes configuration" but this also does not help. How do I access this drive? Or should I take a different approach?

I am trying to walk through the simple ["Submit your first Batch job"](https://us-east-2.console.aws.amazon.com/batch/home?region=us-east-2#firstRun) for AWS Batch. I go through all of the 5 steps, choosing Fargate as the platform, and leaving all items at default values, and finally submit the job. However, the job fails with this status reason: CannotPullContainerError: inspect image has been retried 5 time(s): failed to resolve ref "docker.io/library/amazonlinux:latest": failed to do request: Head https://registry-1.docker.io/v2/library/amazonlinux/manifests/latest: dial tcp 34.237.244.67:44... I am probably missing something very basic, but have no idea what. Thanks!

If I subsmit a Batch array job, what is the best practice for getting the total number of array workers within each array worker itself? I can access the worker index using the `AWS_BATCH_JOB_ARRAY_INDEX` environment variable, but often this number doesn't mean anything (e.g., for determining the extent of a chunk of data for the worker) without the context of the total number of workers. I understand that I can supply the total array size as a command line argument to the job, but this is cumbersome. Is there a better way?

I'd like to have the size of a Batch job be configurable from inputs of a Step Function. However, both: ``` "ArrayProperties": { "Size": "$.ArraySize" } ``` and ``` "ArrayProperties": { "Size": $.ArraySize } ``` do not work. The former complains that type, string, is invalid for the paramter, and the latter complains of a syntax error. Is it what I'd like to achieve possible?

S3 Batch Operations jobs stays "Active" for ever. It has accessed the manifest and shows the number of objects as expected. But after that it doesn't trigger the lambda nor fails with any response. Any help much appreciated

I'm considering using Batch as a more flexible alternative to Lambda where I could run longer or faster processes. A big difference with Lambda is the startup time. I searched and I found Batch startup time can vary [from minutes to hours](https://forums.aws.amazon.com/thread.jspa?threadID=245225&start=25&tstart=0)! Considering my jobs are initiated by the user and the result is expected within minutes. How can the startup time be optimized? Yesterday night the jobs using default (optimal + spot) and no GPU requirements were starting promptly. This morning I can't make any job to start regardless of the machine. Surely there must be some guidance on the matter. Thanks!

Hello, I wonder if there are plans to add support for AWS G5 instances for AWS Batch? They are much better than G4dn instances and there is much more availability compared to G4dn instances. I often have to wait the entire day to get access to a single G4dn instance in `us-east-1` and the same problem is starting to occur in `us-west-2`.

I'm running Fargate in AWS Batch. If I want to give the task S3 access, which of the following Roles should I add the policy to? - AWS Batch execution IAM role - job role I know that Fargate uses task roles. However, there was no setting item called task role in AWS Batch. Best regards,

Following was my code: var jobRequest = new SubmitJobRequest(); jobRequest.Tags = new Dictionary<string, string> {{"test", "testTagValue"}}; response = client.SubmitJobAsync(jobRequest).Result; var response = client.ListJobsAsync(listJobsRequest).Result;//can't read by tags, only can by JobID Is there any AWS Batch API to retrieve the job by tag?

I am trying to implement an Multi-label Image classification model in AWS Rekognition. I used manifest file to create the dataset in AWS Rekognition, but the Rekognition console cannot take the dataset from s3 bucket. while creating the dataset, the console has following errors: 1. Amazon Rekognition custom labels cannot get the dataset entries. Try refreshing the console. 2. The manifest file contains too many invalid rows. I have created the manifest file using the link: https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/md-create-manifest-file-classification.html I followed the steps as described in the following link to create the dataset in AWS Rekognition: https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/md-create-manifest-file.html#create-dataset-procedure-manifest-file The s3 bucket has two folders 1. Train - storing my training images 2. Manifestfiles - storing the manifest file Any help will be appreciated.

I have a CUDA application that requires version 470.x.x of NVIDIA's CUDA drivers. The Amazon Linux 2 ECS-optimized GPU AMI was updated a few weeks ago to carry driver version 470.57.02 (updated from 460.73.01), which is great. However, I find that a new Batch compute environment configured for p3-family instances launches instances using the older AMI amzn2-ami-ecs-gpu-hvm-2.0.20210916-x86_64-ebs from September, which has the old 460.73.01 driver version. This indicates that Batch does not directly track the latest recommended ECS-optimized GPU AMI version. When can I expect Batch to be updated to use the new amzn2-ami-ecs-gpu-hvm-2.0.20211120-x86_64-ebs ECS-optimized GPU AMI with NVIDIA driver version 470.57.02? In general, does AWS have a policy (official or unofficial) for when the ECS-optimized GPU AMI should be updated to include new drivers from NVIDIA, or for when Batch will start using a new ECS-optimized GPU AMI by default? Knowing this would be very helpful for my planning in order to avoid driver version incompatibility issues in the future. Thanks.

I run the command ``` aws batch delete-compute-environment --compute-environment batch-environment ``` It returns with no other messages but the compute environment still persist.

I am able to run ECS tasks and access secrets from Secrets Manager to pass as environment variables to my container. I am attempting to do the same thing with my AWS Batch job definition. 1. I am using the same ecsTaskExecutionRole on both ECS and Batch, so I know the permissions are good for accessing the desired secret because it works on ECS. I triple checked the permissions per https://docs.aws.amazon.com/batch/latest/userguide/execution-IAM-role.html 2. I am almost certain it's not a networking issue, as I'm running the job with a public IP and in a public subnet of my VPC (the same VPC subnet that I have ECS tasks successfully fetching secrets in) 3. When attempting to run a job, I get the following error: "statusReason": "ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secrets from ssm: service call has been retried 1 time(s): AccessDeniedException: User: arn:aws:sts::XXXXXXXX:assumed-role..." 4. If I remove the secrets from the job definition the container will pull from the registry. Both the AWS Web Console and CLI have the truncated statusReason error, which is frustrating because there may be useful info there.

Hi, I have multiple AWS accounts in which I use AWS Batch to run jobs. All Batch compute environments were created with the default EC2 AMI: ECS_AL1 (Amazon Linux). Here's part of the output from _aws batch describe-compute-environments_ for one of my accounts: "type": "MANAGED", "state": "ENABLED", "status": "VALID", "statusReason": "ComputeEnvironment Healthy", "computeResources": { "type": "EC2", "minvCpus": 0, "maxvCpus": 64, "desiredvCpus": 0, "instanceTypes": \[ "c4.large", "c4.xlarge", "c4.2xlarge", "c4.4xlarge", "c4.8xlarge" ], "subnets": \[ "...", "..." ], "securityGroupIds": \[ "..." ], "instanceRole": "arn:aws:iam::...", "tags": {}, "ec2Configuration": \[ { "imageType": "ECS_AL1" } ] In the past I could use yum, as you do in Amazon Linux, inside Batch jobs to install dependencies for my workloads. Yesterday I noticed that yum doesn't work anymore. I also noticed that the OS in my compute environments changed! Now it's Debian. Here's the output of _cat /etc/os-release_ which I ran inside one of my Batch jobs: PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" Why did the OS inside my Batch jobs change and how can I make it Amazon Linux again? For more context, I don't start Batch jobs directly. Metaflow (<https://metaflow.org/>) in a SageMaker Notebook instance does that for me.

Hello, we had a working setup, which unexpectedly stopped working: deep learning containers on AWS Batch. Our stack consisted of: - Compute environments with p3.2xlarge and g4.2xlarge instances - Amazon ECS-optimized GPU AMI (late 2020 and 2021 versions) - Docker base image: nvcr.io/nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04 Our container has a simple nvidia-smi check. This setup worked throughout February, yesterday we ran some jobs on P3 instances, which started failing. We tried running them on G4, they work as expected.. Since nvidia-smi fails, there must be some sort of CUDA or NVIDIA driver mismatch. Regardless, the AMI IDs and base docker versions have not changed between our tests. The P3 instance itself has the following NVIDIA configuration: NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 We tried the updated AMIs but the issue persists. Any ideas what might be causing these issues?

My batch jobs depend on increasing the disk space available to docker by setting --storage-opt in the launch template. According to https://aws.amazon.com/premiumsupport/knowledge-center/increase-default-ecs-docker-limit this setting actually only applies to Amazon Linux 1 AMIs, which will soon be deprecated: _Important: This article is applicable if you're using Amazon Linux 1 AMIs to launch container instances in your ECS cluster. Amazon Linux 2 AMIs use the Docker overlay2 storage driver, which gives you a base storage size of the space left on your disk. Amazon Linux 1 will no longer be supported after December 31, 2020._ I guess this means I am using a v1 AMI; how then do I select a v2 AMI? I have never manually selected an AMI, I always use the defaults; in which case I am surprised that a soon-to-be-deprecated AMI is still the default.

Is it possible to use an Access Point reference instead of a bucket in the manifest for an S3 Batch Job? I have tried a few options but it looks like, since the URL structure is different for Access Points, that Batch can't be configured to use them as a source target.

Hi, In compute env. details I have a message: "CLIENT_ERROR - User: batch.amazonaws.com is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::269363578156:role/tda-sys-app-BatchAWSBatchServiceRole-17ML9A4DFX43" Thus I cannot delete the compute environment neither Cloudformation rollback can.

Since today I have been seeing several failures on batch jobs due to the following error: ``` CannotPullContainerError: failed to register layer: Error processing tar file(exit status 1): write /opt/conda/lib/libmkl_avx512.so: no space left on device ``` I am using the standard AWS ECS AMI, and my compute environment is set up to use 'm4 family' so I would expect it to choose an instance that has enough resources. Is there anything I can do to prevent this?

Hey, I&#39;m running my code on aws batch and after finishing code similar to below, it hangs. ``` with Pool(processes=6) as pool: pool.starmap(self.some_function,[[...] for ind,x in enumerate(list)]) ``` When running it on my computer, there is no problem. it seems that the configurations on my machine and on batch are the same. Is there a known problem about running multiple processes on ecs?

Hi Folks, I'm trying to run a userdata script to mount an EBS volume on my Batch instances during it's launch however no matter what I do the aws cli can't find the IAM Profile credentials. Running aws cli always returns **"Unable to locate credentials. You can configure credentials by running "aws configure".** If I ssh into the instance I can curl the local metadata endpoint and confirm that the the profile were properly assigned. I saw some old posts and blogs about this error but none of the solutions proposed seems to work. Some posts mentions that if the role is attached after the instance launch than aws cli won't see the credentials but I can't say if thats the case or not if instances managed by AWS Batch, nor do I found who should I setup my cloudformation template in order to make the credentials available for the userdata script. Can someone advise me? Has anyone already faced this problem?

Hi folks, I have some ETL jobs that is already containerized and as of today we do all orchestration by our self's in a kind of "in house made aws-batch version". I'm considering to move to AWS Batch but in order to do so I would like to have each Job to run on a dedicate instance due to disk space requirements. By sharing an instance we risk running out space in the middle of the job execution. From what I could see in terms of how AWS Batch handle compute requirements the decisions on when to launch a new instance is based solely on vCpu and Memory. Is there a way to instruct Batch to never send two jobs to the same running instance? Thanks in advance,

BEST_FIT_PROGRESSIVE works weird, maxiumum vcpus = 4, gave me 96. Essentially asked for for 4cpus, m5.xlarge, gave me m5.xlarge24. The auto scaling group tried to spin up 4 instances for 1 job, assuming the other 3 instances were also m5.xlarge24s

Hi, We migrated a few of our scheduled jobs to run aws batch and i have a few questions about the behaviour that i cant find online. The auto scaling groups provisions m4 instances instead of m5s? which would be cheaper, is there a reason for this? The allowed instances is on "optimal" which indicates this(i'll admit i didnt read this at the time of creating the environment). The other question is, whenever it scales up an instance, it keeps it alive for a minimum of 10 minutes even if there are no jobs in a runnable status, is this intended? I read that at the end of the ec2 billing cycle it will check if there are no jobs that are runnable and then scale down, which is weird to me because i read that ec2 is per second billing? Kind Regards and thanks in advance

1) Following https://docs.aws.amazon.com/batch/latest/userguide/create-batch-ami.html, it says to stop the Amazon ECS container agent w/ "sudo stop ecs". That throws an error, i assume you meant "sudo systemctl stop ecs"? 2) I need the image to do some setup before it starts accepting containers. I understand that User Data is suggested for this (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/example_user_data_scripts.html). However i found creating a custom systemd service and installing/enabling it before creating the custom AMI is a much cleaner solution. I haven't found much documentation on creating custom systemd on ecs-optimized Amazon Linux 2. The created image does correctly run the custom systemd service. Is there any potential problems with this approach instead of using User Data?

Hello, I have a working managed ComputeEnvironment and I would like to add "ECS_IMAGE_PULL_BEHAVIOR=prefer-cached" into /etc/ecs/ecs.config in order to assure that the container is not reloaded from ECR for each job submit. I have tried with the following LaunchTemplate which is defined among other Resources in my cloudFormation file: "mwmLaunchTemplate": { "Type" : "AWS::EC2::LaunchTemplate", "Properties" : { "LaunchTemplateData" : { "UserData" : { "Fn::Base64" : { "Fn::Join" : \["", \[ "Content-Type: multipart/mixed; boundary=\"==MYBOUNDARY==\"\n", "MIME-Version: 1.0\n", "\n", "--==MYBOUNDARY==\n", "Content-Type: text/x-shellscript; charset=\"us-ascii\"\n", "MIME-Version: 1.0\n", "Content-Transfer-Encoding: 7bit\n", "\n", "#!/bin/bash\n", "echo \"ECS_IMAGE_PULL_BEHAVIOR=prefer-cached\" >> /etc/ecs/ecs.config\n", "\n", "--==MYBOUNDARY==--" ]]}} } } } The stack creation fails (rolling back without any message) if I am using mwmLaunchTemplate inside of my ComputeEnvironment: "mwmComputeEnvironment": { "Type": "AWS::Batch::ComputeEnvironment", "Properties": { "Type": "MANAGED", ... ... "launchTemplate": { "LaunchTemplateId" : {"Ref": "mwmLaunchTemplate"} }, } .... } mwmComputeEnvironment works well without the three lines: "launchTemplate": { "LaunchTemplateId" : {"Ref": "mwmLaunchTemplate"} }, Thanks for your attention Walter Steffe

Here is an example of a BATCH setup. ``` sampleBatch: Type: AWS::Batch::ComputeEnvironment Properties: State: ENABLED Type: MANAGED ServiceRole: !GetAtt awsBatchServiceRole.Arn ComputeResources: MaxvCpus: 20 MinvCpus: 0 SecurityGroupIds: - !Join [ ",", !Ref "securityGroups" ] Type: EC2 ImageId: !Ref batchAmiImage InstanceRole: !GetAtt ecsInstanceProfile.Arn InstanceTypes: - m4.large Ec2KeyPair: !Ref ec2KeyPair Subnets: !Ref subnets Tags: Name: "Testing Batch Env" sampleBatchJobQueue: Type: "AWS::Batch::JobQueue" Properties: ComputeEnvironmentOrder: - Order: 1 ComputeEnvironment: !Ref sampleBatch State: ENABLED Priority: 1000 sampleCnvBatchJobDefinition: Type: "AWS::Batch::JobDefinition" Properties: Type: container ContainerProperties: JobRoleArn: !Ref ecsTaskRole Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${dockerCnvCallerRepository}:latest" Vcpus: 2 Memory: 2000 MountPoints: - ReadOnly: false SourceVolume: genomics ContainerPath: /genomics Volumes: - Host: SourcePath: /genomics Name: genomics ``` When I submit multiple jobs, they get queued. What I want to achieve is that Batch start new instance for each job. I cannot find a combination of settings that will allow me to do this. Any advice welcome.

In AWS Batch, sometimes the job is stuck in RUNNABLE state. Under what conditions the job gets stuck? Is there a way to find out the reason job gets stuck? Also Is there a way to alert via Cloudwatch metrics and SNS? Also is there a max time after which if job is still stuck, it means that the job will never run?? Edited by: arian on Mar 17, 2019 8:38 AM

Hello I have many jobs that should roughly take about 10mins. Initially, on a fresh ec2 instance they do take 10 mins, but as that instance is reused for hours, jobs start taking 30mins. When I increased the compute environment size and new instances are created, their jobs are back down to 10 mins. When looking at old vs new instances, the new instances use about 90% of the cpu, while the older on is maybe 20%. Any idea, what is going on? and how I can fix this without shutting down running instances?

Hi, I see on AWS batch help page somewhere, it was mentioned that we can provide a our application as a zip file in a job and our application will be directly executed inside an EC2 instance without the need of a docker instance. I have not seen a single example of such kind on internet. Is anyone aware if there is a way to use AWS batch without using a docker image? Definition of Job from AWS official documentation: A unit of work (such as a shell script, a Linux executable, or a Docker container image) that you submit to AWS Batch. - Thanks, Sid Edited by: Siddharthvij on Nov 25, 2018 11:29 PM

Does AWS Batch have support for EC2 spot stop/start and hibernation options to handle interruptions? The options are documented here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html But I am unable to find reference of whether AWS Batch can support this. From my understanding, if a Spot instance is terminated, AWS Batch retries the job from the start.

Customer is asking if their Batch job that execute on spot is interrupted due to spot price increasing beyond their threshold, will Batch resubmit the job when the spot price falls again, or will the customer have to manually resubmit?