Batch queues and jobs quotas

Question

Dear all,
My questions are very basic.
I have created a compute env and a job queue in the ca-central-a region, with a maximum of vCPU=100.

(1) When I send 200 jobs to this queue, each job being fully independent and requiring only 1 vCPU and 2G RAM, only 5 jobs run, the others waiting as runnable. I would expected dozens of such small jobs to be able to run in parallel

(2) The number of jobs I have is now about 12k, which makes a big list, is there a way to delete old jobs and purge the list?

Thanks in advance,
regards,
Xavier

Answer

Hello!

(1) In AWS Batch, the RUNNABLE jobs are started as soon as sufficient resources are available in one of the compute environments that are mapped to the job's queue. However, AWS Batch is a scheduler and optimizes for cost performance, job priority, and throughput. AWS Batch might require time to process your requests.

Additionally, the AWS Batch Scheduler periodically evaluates jobs in the queue and moves them forward as appropriate. As you submit more jobs, you will see that the AWS Batch Scheduler evaluates and operates upon many jobs at once with each scheduling interval.

(2) Jobs can be terminate using TerminateJob API or CLI call. Jobs that are in the STARTING or RUNNING state are terminated, which causes them to transition to FAILED . Jobs that have not progressed to the STARTING state are cancelled.
https://docs.aws.amazon.com/cli/latest/reference/batch/terminate-job.html

Batch queues and jobs quotas

Relevant content