AWS Batch Compute Environment never launches EC2 instances (both EC2 and Managed Instances modes) — jobs stuck in RUNNABLE, zero Auto Scaling Groups created

Hello.

Have you checked CloudTrail for the "CreateAutoScalingGroup" event?
It's possible that the AutoScaling Group creation failed for some reason.
Please check the logs using the following commands or the management console.
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events-cli.html

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateAutoScalingGroup

AWS Batch Jobs Stuck in RUNNABLE — Troubleshooting Guide Given everything you've validated, the issue is that Batch is trying to schedule tasks but never provisions EC2 instances. Here are the most likely root causes in order of probability:

1. Missing or Incorrect Service-Linked Role for Batch (Most Likely) AWS Batch needs the service-linked role AWSServiceRoleForBatch to manage EC2 instances/ASGs on your behalf. This is separate from the roles you've configured.

Check:

aws iam get-role --role-name AWSServiceRoleForBatch --region us-east-2

If it doesn't exist, create it:

aws iam create-service-linked-role --aws-service-name batch.amazonaws.com

This role grants Batch permission to create/manage Auto Scaling Groups, launch EC2 instances, and manage ECS cluster membership. Without it, Batch silently fails to provision — everything shows "Valid" but no instances appear.

2. Subnet Configuration — No Route to Internet or AWS APIs Batch-managed EC2 instances need access to ECS, ECR, S3, and CloudWatch APIs at launch. If your subnets are private without a NAT Gateway, or have restrictive route tables, instances can't register with ECS even if they launch.

Check: Are the subnets in the Compute Environment private subnets? If yes — do they have a NAT Gateway route? If public subnets — do they have auto-assign public IP enabled? Do the route tables have a path to the internet (0.0.0.0/0 → NAT or IGW)? Alternatively, ensure VPC endpoints exist for:

com.amazonaws.us-east-2.ecs
com.amazonaws.us-east-2.ecs-agent
com.amazonaws.us-east-2.ecr.api
com.amazonaws.us-east-2.ecr.dkr
com.amazonaws.us-east-2.s3 (gateway)
com.amazonaws.us-east-2.logs

3. Security Group Blocking Outbound Traffic The security group assigned to the Compute Environment must allow outbound HTTPS (port 443) to reach AWS APIs. Without this, instances launch but fail to register with ECS → they get terminated → cycle repeats (or they never launch at all).

Check:

aws ec2 describe-security-groups --group-ids {security_group_id} --query "SecurityGroups[0].IpPermissionsEgress"

Must have at minimum:

Port 443 outbound → 0.0.0.0/0 (or VPC endpoint prefixes)

4. EC2 Service Quota for g6.xlarge You confirmed you can manually launch a g6.xlarge — but Batch uses a different quota than on-demand console launches. Check the Running On-Demand G and VT instances service quota:

aws service-quotas get-service-quota \
  --service-code ec2 \
  --quota-code L-DB2F2B5E \
  --region us-east-2

The default for G instances might be 0 vCPUs in some accounts/regions. Even if you can manually launch one via console (which sometimes works under a different quota path), Batch's ASG-based provisioning is strictly governed by this quota.

5. Launch Template or AMI Issue If a launch template is associated with the Compute Environment:

Verify the AMI ID is valid and exists in us-east-2 Check that it's an ECS-optimized GPU AMI (for g6, you need the Amazon Linux 2 GPU-optimized AMI) Ensure the launch template doesn't have conflicting settings (wrong instance type, wrong subnet, etc.) If no launch template — Batch should auto-select the ECS-optimized AMI. But verify:

aws ssm get-parameters --names /aws/service/ecs/optimized-ami/amazon-linux-2/gpu/recommended --region us-east-2

6. Availability Zone Capacity g6.xlarge (powered by NVIDIA L4) may not be available in all AZs within us-east-2. If your subnets are limited to AZs without g6 capacity, Batch can't provision.

Check: Ensure the Compute Environment subnets span multiple AZs, or test by specifying subnets in AZs where you successfully launched manually.

You've done exceptional diagnostic work here — you've genuinely ruled out capacity, quota, IAM, networking, and AMI. The single most important clue in everything you've posted is this:

Batch calls ecs:RunTask continuously, but CloudTrail shows CreateAutoScalingGroup is never called ({"Events": []}), and no ASG ever exists.

That combination tells us that Batch is behaving as if capacity should already exist, so it keeps trying to place tasks — but the provisioning half of the service (the part that creates the ASG and scales it) is never engaging. Due to this we do not see failed provisioning attempts, rather we see no provisioning attempts.

Let's try few more methods of diagnosis.

1.

aws batch describe-compute-environments \
  --compute-environments chemtox-g6-compute-v2 \
  --region us-east-2 \
  --query "computeEnvironments[0].computeResources.{min:minvCpus,desired:desiredvCpus,max:maxvCpus,type:type,subnets:subnets,instanceTypes:instanceTypes}"

Our goal is to Check the compute environment's minvCpus, desiredvCpus, and maxvCpus.

• If maxvCpus is 0 (or lower than a single g6.xlarge's 4 vCPUs), Batch cannot scale up at all and will never create/expand the ASG. This is easy to set accidentally.
• Confirm instanceTypes actually includes g6 / g6.xlarge (or an appropriate family). A GPU job routed to a CE whose instance list resolves to no GPU-capable type will sit in RUNNABLE forever.
• Confirm the CE type is MANAGED, not UNMANAGED. An unmanaged CE never creates an ASG or provisions anything — you'd be responsible for launching and registering instances yourself, which would perfectly explain "0 container instances, no ASG, RunTask loop." Given you expected Batch to auto-create the ASG, verify this field explicitly.

2. Confirm the job's resource requirements can ever fit

Your GPU job definition must request a GPU resource so Batch selects a GPU instance:

resourceRequirements:
  - type: GPU
    value: "1"
  - type: VCPU
    value: "4"
  - type: MEMORY
    value: "14000"

If the job requests more vCPU/memory than a g6.xlarge can offer (after ECS agent overhead), or requests a GPU the CE's instance types can't supply, the job never becomes placeable — and Batch won't scale a CE it believes can't satisfy the job.

3. Rule out a leftover / blocking job at the head of the queue

Submit a trivial CPU-only job (256 MB, 1 vCPU, no GPU) to the same queue. If even that never provisions, the problem is squarely the CE/ASG wiring, not GPU-specific. If the trivial job does provision an instance, the issue is your GPU job's resource requirements or GPU AMI selection.

4. Verify the service-linked role can actually create the ASG

You confirmed AWSServiceRoleForBatch exists — but for managed EC2 CEs, Batch uses this SLR to call autoscaling:CreateAutoScalingGroup, ec2:RunInstances, iam:PassRole (passing your ecsInstanceRole), and ec2:CreateLaunchTemplate. A denied iam:PassRole or a missing instance profile can silently prevent ASG creation. Check CloudTrail for RunInstances and PassRole events (not just CreateAutoScalingGroup) and look for AccessDenied.

5. Managed Instances vs classic EC2

Note that "ECS Managed Instances" is a newer, distinct compute model from classic managed-EC2 Batch. Since both modes fail identically for you, that actually strengthens the case for an account- or CE-level configuration issue (most likely maxvCpus/type, or a region-specific setup problem in us-east-2) rather than a bug in one mode.

My strong bet: run the command in step 1 first. A maxvCpus of 0, an UNMANAGED CE type, or an instance-type list that doesn't include a GPU family is by far the most likely explanation for an ASG that is never created while RunTask loops — and it fits every single symptom you've documented. Happy to look at the exact JSON output if you post it.