How do I troubleshoot an AWS Batch job that's stuck in the RUNNABLE status?

Short description

AWS Batch moves a job to the RUNNABLE status when the job has no pending dependencies and AWS Batch can schedule the job. RUNNABLE jobs start as soon as sufficient resources are available in one of the compute environments that map to the job's queue.

If the required resources to run a job aren't available, then the job might remain stuck in the RUNNABLE status.

Note: The following resolution troubleshoots AWS Batch jobs that run in Amazon Elastic Container Service (Amazon ECS) containers. These containers can run on either Amazon Elastic Compute Cloud (Amazon EC2) instances or AWS Fargate compute environments. To troubleshoot AWS Batch jobs that run on Amazon Elastic Kubernetes Service (Amazon EKS), see AWS Batch on Amazon EKS.

Resolution

First, determine whether your stuck job is blocking the first-in, first-out (FIFO) queue. If this doesn't resolve your issue, then either automate the troubleshooting to determine the issue or manually troubleshoot the issue.

Check for a blocked FIFO queue

A job might be stuck in the RUNNABLE status at the beginning of a first-in, first-out (FIFO) queue because of the following issues:

• The job doesn't use fair-share scheduling, so all other jobs behind it can't run.
• There's a misconfiguration in your AWS account.
• Your account doesn't have access to the required instances for the job. For example, your account might require a GPU instance type.

To unblock the FIFO queue, use GetJobQueueSnapshot to find the job that's blocking the queue, and then shut down the job.

Automate the troubleshooting process

Use the AWSSupport-TroubleshootAWSBatchJob runbook to troubleshoot an AWS Batch job that's stuck in the RUNNABLE status. Or, complete the steps in the following sections to manually troubleshoot the issue.

Check whether your compute environment has enough resources to run your job

Complete the following steps:

1. Open the AWS Batch console.
2. In the navigation pane, choose Dashboard.
3. In the Job queue overview section, in the RUNNABLE column, select the job that's stuck in the RUNNABLE status. The details page for your job appears.
4. Choose the Containers tab, and then note the values for vCPUs, Memory, and GPUs to complete steps 9–10.
5. In the navigation pane, choose Job queues, and then choose your job queue.
6. On the Environment order tab, identify the compute environments that are associated with your job queue.
7. In the navigation pane, choose Environments, and then select a compute environment to review its permissions.
8. In the Status section, confirm that the compute environment's Status column is Valid.
   Note: When there are intermittent or transient errors, it might take a few minutes for the compute environment status to change from Valid to Invalid.
9. On the Details tab, verify that the service role that's associated with the environment has all the necessary permissions.
10. In the Status section, verify that the State column is Enabled.
11. On the Compute resources tab, check the Maximum vCPUs value. This value must be high enough so that AWS Batch can increase the number of Desired vCPUs to run jobs.
    Note: If you use a Fargate compute environment, then see the Check the network and security settings of the compute environment section.
12. Verify that the Desired vCPUs value is the same or higher than the number of vCPUs that the job must run.
13. If Desired vCPUs is 0, then check the amount of memory and CPU resources that are available for your Amazon EC2 instance type. If Desired vCPU is higher than 0 or your job is still in the RUNNABLE status, then complete the steps in the following section.

Repeat steps 6–14 for each compute environment.

Important: At least one of the instance types for your compute environment must have more memory than what your job specifies. Also, the instance type must have CPU resources that are equal to or more than what your job specifies. If at least one instance type doesn't have enough memory or CPU resources to run your job, then cancel the job. Run a new job that requires less CPU or memory. Or, create a new compute environment with enough resources to run the job, and then assign the job to the appropriate job queue.

Verify that your compute environment has instances and the instances are available to run your job

For the compute environment that's running your job, complete the following steps:

1. Open the Amazon ECS console.
2. In the navigation pane, choose Clusters, and then select the cluster that contains your job.
   Note: The name of the cluster starts with the name of the compute environment followed by _Batch_ and a random hash of numbers and letters.
3. Choose the Infrastructure tab.
4. In the Container instances section, find your container instances, and then verify that your container instances are available to run your job.

If the cluster has an available container instance to run your job, then check the status of the Docker daemon and Amazon ECS container agent. For more information, see How do I troubleshoot a disconnected Amazon ECS agent?

If there are no instances in the Amazon ECS cluster, then check whether you can create instances in your compute environment.

On-Demand compute environment

Complete the following steps:

1. Open the Amazon EC2 console.

2. In the navigation pane choose Auto Scaling, and then choose Auto Scaling Groups.

3. In the search box, enter the name of your compute environment, and then select your compute environment.
   Note: Amazon EC2 can create more than one Auto Scaling group for the same compute environment.

4. For each Auto Scaling group, choose the Activity tab, and then look for blocking issues in the Activity history section.
   Note: The Status column shows Unsuccessful when there are issues that cause instances not to launch.
   For example, if your account reaches the maximum number of instances, then Amazon EC2 might return a message that's similar to the following one:
   "Launching a new EC2 instance. Status Reason: Your quota allows for 0 more running instance(s). You requested at least 1. Launching EC2 instance failed."
   The event includes a timestamp in UTC from when you submitted the job:

   At 2018-09-03T05:54:30Z a user request update of AutoScalingGroup constraints to min: 0, max: 1, desired: 1 changing the desired capacity from 0 to 1.At 2018-09-03T05:54:52Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1.

   Note: AWS Batch requests instances on your behalf. If you manually modify the Auto Scaling groups, then your compute environment might be invalidated. For more information about instance quotas and how to request a quota increase, see Amazon EC2 service quotas.

If the Auto Scaling group shows only successful events in Recent events, then complete the steps in the Check the container instance's IAM role section.

Important: You must set permissions for the AWSServiceRoleForAutoScaling AWS Identity and Access Management (IAM) service role. The AWSServiceRoleForAutoScaling IAM role must have access to the customer managed AWS Key Management Service (AWS KMS) key. This is necessary in environments with custom Amazon Machine Images (AMIs), encrypted Amazon Elastic Block Store (Amazon EBS) volumes, and customer managed AWS KMS keys. For more information, see Key policy sections that allow access to the customer managed key.

Spot compute environment

Complete the following steps:

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Spot Requests.
3. Choose Request type, and then choose Request type = fleet.
4. Select your spot request.
5. For Status, choose Active.
6. Choose Description, and then review the Total target capacity value to determine whether the Spot Instance request was fulfilled. If there's no instance, then check the History view to determine why.
   For example, requests that can't reach a bid price return a message that's similar to the following one:
   "m4.large, ami-aff65ad2, Linux/UNIX (Amazon VPC), us-east-1a, Spot bid price is less than Spot market price $0.0324"
7. Choose an appropriate bid percentage for your compute environment. If you change the bid price, then make sure that you create a new compute environment. For more information, see Spot Instance pricing history.
   Note: AWS Batch creates Spot Fleet requests on your behalf. Don't manually modify Spot Fleet requests, or your compute environment might be invalidated.

If the most recent events of the Auto Scaling group show only successful events, then complete the steps in the following section.

Check the container instance's IAM role

Complete the following steps:

1. Open the AWS Batch console.

2. In the navigation pane, choose Environments, and then select your compute environment.

3. On the Details tab, note the Instance role name.

4. Open the IAM console.

5. In the search box, enter the Instance role name, and then select your instance role.

6. Choose the Permissions tab. In the Permissions policies section, confirm that you attached the AmazonEC2ContainerServiceforEC2Role managed policy to the role. If you attached the policy, then proceed to step 11.

7. Choose Add permissions, and then choose Attach policies.

8. In the Other permissions policies section, enter AmazonEC2ContainerServiceforEC2Role in the search box.

9. Select AmazonEC2ContainerServiceforEC2Role, and then choose Add permissions.

10. Choose the Trust relationships tab, and then choose Edit trust policy.

11. Confirm that the trust policy contains the following statement:

    {  "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "",
          "Effect": "Allow",
          "Principal": {
            "Service": "ec2.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }

12. If the trust policy includes the preceding statement, then choose Cancel. If the trust policy doesn't include the preceding statement, then add the statement to your policy. Then, choose Update policy.

If your instance still doesn't join the Amazon ECS cluster, then complete the steps in the following section.

Check the network and security settings of the compute environment

Complete the following steps:

1. Open the AWS Batch console.
2. In the navigation pane, choose Environments, and then select your compute environment.
3. In the Compute resources section, note the Subnets and Security groups values.
4. Open the Amazon Virtual Private Cloud (Amazon VPC) console.
5. In the navigation pane, choose Subnets.
6. Select each subnet in the compute environment, and then identify the Auto-assign public IPv4 address values on the Details tab.
   If the Auto-assign public IPv4 address value is Yes, then the instances that launched in the subnet have the following properties:
   A public IPv4 address
   A route table with a route destination of 0.0.0.0/0
   An internet gateway, for example igw-1a2b3c4d, that's set to Target
   If the Auto-assign public IPv4 address value is No, then the instances that launched in the subnet have the following properties:
   A private IPv4 address
   A route table with a route destination of 0.0.0.0/0
   A NAT gateway, for example nat-12345678901234567, that's set to Target
   Note: For more information, see Routing.
7. In the navigation pane, choose Security groups.
8. For each security group in the compute environment, choose the Outbound rules tab. Then, verify that a rule exists with the following settings:
   For Type, choose All traffic.
   For Protocol, choose All.
   For Port range, choose All.
   For Destination, choose and 0.0.0.0/0.
   Important: If the rule doesn't exist, choose Actions, and then choose Edit outbound rules. Then, create the rule. For a more restrictive rule for outbound traffic, choose HTTPS (443) for Type and 0.0.0.0/0 for Destination.
9. In the navigation pane, choose Network ACLs.
10. Select the VPC's network access control list (network ACL).
11. On the Inbound rules and Outbound rules tabs, confirm that the network ACL allows all traffic to flow in and out of associated subnets.

Important: If you modified the network ACL, then add a rule that allows outbound IPv4 HTTPS traffic from the subnet to the internet. For more information, see Control traffic to your AWS resources using security groups and Control subnet traffic with network access control lists. To change the VPC, subnets, or security groups, create a new compute environment.

If your instance still doesn't join the Amazon ECS cluster, then connect to your instance. Check the status of the Docker daemon and Amazon ECS container agent. For more information, see How do I troubleshoot a disconnected Amazon ECS agent?

Note: To further troubleshoot an AWS Batch job that's stuck in the RUNNABLE status, use AWS CloudTrail. Set the Username attribute to aws-batch to search for errors that occur during scheduled tasks.

Related information

Connect to your Linux instance using SSH

Connect to your Windows instance using RDP

Introducing new alerts to help users detect and react to blocked job queues in AWS Batch

AWS Batch introduces the Job Queue Snapshot to view jobs at the front of the job queues