How to get a list of ALL SQS queues?

Question

For various reasons, the number of our SQS queues has grown tremendously. We'd like to get a list of all the existing queues. Using AWSCli, we are bounded by the upper limit of 1000 at most. Is there a way we can retrieve a complete list of ALL sqs queues? Also, is there a regex of "negative" match with --queue-name-prefix? Could you please provide some examples? Thank you so much!

Accepted Answer

Ah, this one was tricky.  I had to create 1005 queues to figure it out.  The 1000 limit appears to be related to the max number of results that will fit in a request made by the client(aws cli itself).  When working with more than 1000 queues it looks like you need to adjust both `--max-items` and `--page-size` to get through.  The `--page-size` will allow the client to make more requests behind the scenes.  Here is how I got through it, your results may vary.
```bash
$ aws sqs list-queues --max-items 1000 --page-size 1000
{
    "QueueUrls": [
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-0001",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-0002",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-0003",
...
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-0998",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-0999",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1000"
    ],
    "NextToken": "...TQT09In0="
}
```
I had to use the `--max-items` to get the NextToken to show up...

Then to get the additional queues: 
```
$aws sqs list-queues --max-items 1000 --page-size 1000 --starting-token "...TQT09In0="
{
    "QueueUrls": [
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1001",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1002",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1003",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1004",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1005"
    ]
}
```

Answer

You need to user the ListQueues API. In there you specify the MaxResults. If there are more results than what you asked, you will receive in the response also a NextToken. You should then invoke the API again and specify the value that you received in the NextToken parameter. This is the way to paginate over all the queues (and other AWS resources).

There is no way in the API for negative filters, but I think you can use a tool like JQ to filter the results on the client side (did not try it).

Answer

Here is an example using the paging api on the cli.  In this example there are 25 queues, we are limiting each page to 15.
```
$ aws sqs list-queues --max-items 15 
{
    "QueueUrls": [
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-01",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-02",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-03",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-04",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-05",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-06",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-07",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-08",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-09",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-10",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-11",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-12",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-13",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-14",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-15"
    ],
    "NextToken": "eyJOZXh0VG9rZW4iOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAxNX0="
}
```
That is the first 15 queus, when there are more left it will provide a `NextToken` that you need to include in the next request, like:
```
$ aws sqs list-queues --max-items 15 --starting-token eyJOZXh0VG9rZW4iOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAxNX0= 
{
    "QueueUrls": [
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-16",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-17",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-18",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-19",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-20",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-21",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-22",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-23",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-24",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-25"
    ]
}
```
In the second request, we asked for 15 but there were only 10 left so we don't get a `NextToken` and that tells us we are done.

How to get a list of ALL SQS queues?

Relevant content