Questions tagged with AWS Batch

On AWS batch, the command, `cat /proc/cpuinfo | grep 'model name'` yields `model name : AMD EPYC 7R13 Processor`. However, I want to use `intel` CPU. How to do it? Thankslg...

I'm trying to create a Batch setup in Cloudformation. I have in Resources an IAM Role: ``` SecretsAndS3AccessRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: batch.amazonaws.com Action: 'sts:AssumeRole' - Effect: Allow Principal: Service: ec2.amazonaws.com Action: 'sts:AssumeRole' - Effect: Allow Principal: Service: ecs-tasks.amazonaws.com Action: 'sts:AssumeRole' ManagedPolicyArns: - 'arn:aws:iam::aws:policy/SecretsManagerReadWrite' - 'arn:aws:iam::aws:policy/AmazonS3FullAccess' - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy' ``` Then in my JobDefinition I have: ``` JobDefinition: Type: 'AWS::Batch::JobDefinition' Properties: Type: container ContainerProperties: Image: uri/to/my/image Vcpus: 2 Memory: 2000 Command: - /simple-test Privileged: true JobRoleArn: !Ref SecretsAndS3AccessRole ExecutionRoleArn: !Ref SecretsAndS3AccessRole Secrets: - Name: MY_SECRET ValueFrom: arn:aws:secretsmanager:us-east-1:123456789:secret:MYSECRET-abcdef RetryStrategy: Attempts: 1 ``` When I try to build the stack, I get: > An error occurred (ClientException) when calling the RegisterJobDefinition operation: Error executing request, Exception : executionRoleArn bothrefs-SecretsAndS3AccessRole-1INAOWFBH2SK2 is not an iam role arn If I remove the `ExecutionRoleArn` line and the Secrets, the stack builds fine, which is to say that `JobRoleArn` is happy with a value of `!Ref SecretsAndS3AccessRole`. (But I need the secrets, and to use secrets you need an execution role.) And if I hardcode the ARN there, it works fine. What is different about `ExecutionRoleArn` that it doesn't allow a `!Ref`? According to [the documentation for JobDefinition/ContainerProperties][1], `JobRoleArn` and `ExecutionRoleArn` seem the same sort of object. If I instead use: ``` ExecutionRoleArn: !GetAtt SecretsAndS3AccessRole.Arn ``` Then it works fine! I tested removing JobRoleArn entirely - that makes my job fail. I tested changing it to also be `JobRoleArn: GetAtt SecretsAndS3AccessRole.Arn` -- that succeeds. So the mystery is: `JobRoleArn` likes its value either in Ref or GetAtt form, but ExecutionRoleArn requires GetAtt form. Why the difference? [1]: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-batch-jobdefinition-containerproperties.htmllg...

When creating new AWS Batch jobs via the SubmitJob API (done in Python, boto3), I see the submitted tags show up in the AWS Batch console. Where I _don't_ see them though, is - The DescribeJob API (done via AWS CLI, `aws batch describe-jobs ...` - Job State change events sent to EventBridge Is this a bug or intended behavior? If it's intended, what is the appropriate way to tag my jobs so that Batch events have arbitrary key/value pairs that I can match EventBridge rules against?lg...

Hello, I am trying to upload a **113 MB** (119.244.077 byte) video to my bucket, it always takes **48 seconds**, even if I use TransferConfig, it seems that multythread uploading does not work, any suggestions? ``` def upload_to_s3(file_name, bucket,path_s3): config = TransferConfig(multipart_threshold=1024 * 25, max_concurrency=10, multipart_chunksize=1024 * 25, use_threads=True) try: start_time = time.time() _ = s3_client.upload_file(file_name, bucket, path_s3, Config=config) elapsed_time = time.time() - start_time print(f"Time: {elapsed_time}") except ClientError as e: logging.error(e) return False path_s3 = "something" config = Config(connect_timeout=5, retries={'max_attempts': 0},max_pool_connections=25) s3_client = boto3.client('s3', aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=REGION_NAME, config=config) # Set here the path of the file path_file_to_upl = "./data/80e12098-ec85-59db-6e36-82e04e884439.mkv" #Upload upload_to_s3(path_file_to_upl, BUCKET_NAME,path_s3) ``` With the above code I take **48/49 seconds**, if I set *use_threads=False* the time increases to **71 seconds**lg...

We created an AMI with a preloaded docker image and ECS agent configurations (under */etc/ecs/ecs.config*) in hopes to bring our AWS Batch execution time down. Although the pull time has been eliminated, the job execution time has doubled (from our prod jobs that still pull the docker image). Has anyone experienced this issue or have guidance on how to debug?lg...

I'm trying to execute a Batch Job with the help of AWS AMI. The Compute Environment for the job is intended to have certain installations for which I need to use AMI. But whenever I give the AMI reference in Compute Environment, the job gets stuck in RUNNABLE state forever. I have tried all the troubleshooting measures given in https://aws.amazon.com/premiumsupport/knowledge-center/batch-job-stuck-runnable-status/ but even that did not help. I even tried using different AMIs for the same. It has been observed that jobs get stuck whenever any custom AMI is used,. As soon as job is submitted, the instance is launched but the job still fails to execute. Without the use of AMI, I have been able to execute the job successfully. I want to know the possible causes and reasons of this. Following are the details of my Compute Environment: ``` { "computeEnvironmentName": "computeenvironmentname", "type": "MANAGED", "state": "ENABLED", "unmanagedvCpus": 0, "computeResources": { "type": "EC2", "allocationStrategy": "BEST_FIT_PROGRESSIVE", "minvCpus": 0, "maxvCpus": 256, "desiredvCpus": 0, "instanceTypes": [ "p3.2xlarge" ], "imageId": "imageid", "subnets": [ "list of subnets" ], "securityGroupIds": [ "security groups" ], "ec2KeyPair": "keypair", "instanceRole": "instancerole", "tags": { "KeyName": "" }, "placementGroup": "", "bidPercentage": 0, "spotIamFleetRole": "", "launchTemplate": { "launchTemplateId": "", "launchTemplateName": "", "version": "" }, "ec2Configuration": [ { "imageType": "ECS_AL2_NVIDIA", "imageIdOverride": "imageid" } ] }, "serviceRole": "servicerole", "tags": { "KeyName": "" } } ```lg...

AWS Batch creates auto scaling groups for managed EC2 compute environments. I would like to monitor the auto scaling behavior but these auto scaling groups are created with CloudWatch metrics disabled. I have not found a way to configure a compute environment such that its auto scaling groups get created with **metrics enabled**. Is there a way to do this?lg...

When setting up a managed compute environment, AWS Batch allows to set [tags on compute resources](https://docs.aws.amazon.com/batch/latest/APIReference/API_ComputeResource.html#Batch-Type-ComputeResource-tags) it creates. Per my experiments these tags get only applied to EC2 instances launched in the compute environment, but not to the ECS cluster nor the EC2 auto scaling group created for the environment. Is this a bug in AWS Batch? If not, what's the recommended way to propagate tags to Batch-initiated ECS clusters and auto scaling groups?lg...

I'm trying to understand what could cause AWS Batch jobs (using EC2 spot instances) to be at times seemingly stuck in "RUNNABLE" state for several hours, before finally getting picked up. This annoying behaviour seems to come and go over time. Some days, the very same jobs, configured in the very same way, using the same queues and compute environments are almost immediately picked up and processed, and some days they will be in RUNNABLE status for a long time (recently I experienced 3 to 6 hours). The usual troubleshooting documents don't help, as they seem to only cover cases where the job *never* gets picked up (due to configuration issue, or mismatch of vCPU/memory between job and compute environments). What I observe when I hit these issues is that there doesn't seem to be any spot request shown in the EC2 dashboard. The Spot instance pricing at that time, for the type of instance I need (32 vCPU, 64Gb memory) is not spiking (and I've set it to a limit of 100% of on-demand anyway). So one theory is that there is no spot instance available at all at that time but 1) that seems unlikely (I use eu-west-1) and 2) I can't find any way to validate that theory. My account limits on M and R instance types (the ones typically be used when the jobs are running) is very high (1000+), so that's also not the reason as far as I can tell. Anyone with any theory and suggestion? For now, my solution is to change the queue to add a compute environment with on-demand instances, but that more than doubles the price...lg...

Hello, I have an AWS batch job running on fargate monitored with cloudwatch container insights that gets stuck for more than 12 hours. Locally on my laptop's i9 the exact same job only takes 1h. My job is configured to use 4 vcpus and 30g of RAM. Container insights is not showing the container using anywhere near the max resource usage. It peaks at around 2g of RAM and then gets stuck. I'm trying to troubleshoot if this is specific to AWS batch or maybe a combination of my workload plus batch. Are there any other things I could do to troubleshoot this? Thanks.lg...

I am trying to use a launch template with a AWS Batch Compute Environment, but I keep getting this error: ```INVALID - CLIENT_ERROR - You are not authorized to use launch template: Batch-lt-f51c5077-bb43-3a1a-b8eb-e7b971b2f39b``` I have verified that both the batch service role and the instance profile have the appropriate permissions. I have been unable to find any more information about this error online. The compute environment works fine without a launch template. The launch template that it is referencing is not the launch template I specified, but a copy of it created by the batch service role (I understand this is normal) Can anyone assist me ?lg...

I am using AWS CLI with cloudwatch put-metric-alarm to set up custom memory alarms when over 50%. It is successful at sending alerts when shifting into the Alarm state from OK or Insufficient and when going from Alarm state to OK. However now it is sending alerts at the same rate when shifting from Insufficient state to Okay. Looking into the documentation it states that the Ok actions trigger when the alarm transitions to an OK state from any other state. Is there a way to stop alerts from the Insufficient State -> OK state?lg...