The number of jobs in the Default queue on my AWS Elemental MediaConvert keeps increasing, and I want to pause queue and cancel jobs.
Short Description
Queues can be ACTIVE or PAUSED. If you pause a queue, jobs in that queue won’t begin. If you pause a queue, the service won't begin processing jobs in that queue. You can cancel a job using either the MediaConvert console or CLI.
However, Cancelling thousands of tasks using console is not only time-consuming but also annoying. Therefore, we introduce you to cancel with a simple shell script using the CLI.
The idea is :
- Pause a queue
- Retrieve list of jobs in a queue
- Cancel a job
- Iterate over a list using loop in bash
Steps
1. Pause on-demand queues
To change the status of an on-demand queue by using the MediaConvert console:
- Open the Queues page in the MediaConvert console.
- In the On-demand queues section, select the queue.
- Choose Edit queue.
- Under Status, choose Pause or Active.
- Choose Save queue.
2. Retrieve list of jobs in a queue
$ aws mediaconvert describe-endpoints --region ap-northeast-2
{
"Endpoints": [
{
"Url": "https://bnklbqvoa.mediaconvert.ap-northeast-2.amazonaws.com"
}
]
}
- Use the list-jobs [2] command in MediaConvert to list the job IDs in the SUBMITED state.
aws mediaconvert list-jobs --region ap-northeast-2 --endpoint-url https://bnklbqvoa.mediaconvert.ap-northeast-2.amazonaws.com --status SUBMITTED --queue Default --query 'Jobs[*].{Id:Id}' --output text
3.Cancel a job[3] using cancel-job in MediaConvert
The following cancel-job example cancels a job.
aws mediaconvert cancel-job \
--endpoint-url https://abcd1234.mediaconvert.region-name-1.amazonaws.com \
--id 1234567890123-efg456
4. Iterate list of jobs in a queue using bash script
- Use the list-jobs command to determine how many tasks you need to cancel.
- Then write code to cancel jobs using the bash script in units of 50 or 100.
Sample code :
Ex) This is the sample code written by MAC BOOK that I am using. You must modify it to suit your environment.
(<endpoint url> must include the endpoint output from #1, <regioin, ex. us-east-1> is the same area information as ap-nortaste-2.)
#!/bin/sh
$aws mediaconvert list-jobs --region <regioin, ex. us-east-1> --endpoint-url <endpoint url> --status SUBMITTED --queue Default --query 'Jobs[*].{Id:Id}' --output text | while read id;
do
echo "Read Job id : $id"
aws mediaconvert cancel-job --region <regioin, ex. us-east-1> --endpoint-url <endpoint url> --id $id > /dev/null
echo "Cancel Job id : $id"
done
Additional comments :
Apart from job canceling, I would like to encourage you to have multiple on-demand queues in order to run jobs for different workflows in separate queues.
References :
[1] https://docs.aws.amazon.com/mediaconvert/latest/ug/working-with-on-demand-queues.html#updating-queue-status
[2] https://docs.aws.amazon.com/cli/latest/reference/mediaconvert/list-jobs.html
[3] https://docs.aws.amazon.com/cli/latest/reference/mediaconvert/cancel-job.html