By using AWS re:Post, you agree to the Terms of Use

How to get a list of ALL SQS queues?

0

For various reasons, the number of our SQS queues has grown tremendously. We'd like to get a list of all the existing queues. Using AWSCli, we are bounded by the upper limit of 1000 at most. Is there a way we can retrieve a complete list of ALL sqs queues? Also, is there a regex of "negative" match with --queue-name-prefix? Could you please provide some examples? Thank you so much!

3 Answers
2

You need to user the ListQueues API. In there you specify the MaxResults. If there are more results than what you asked, you will receive in the response also a NextToken. You should then invoke the API again and specify the value that you received in the NextToken parameter. This is the way to paginate over all the queues (and other AWS resources).

There is no way in the API for negative filters, but I think you can use a tool like JQ to filter the results on the client side (did not try it).

profile picture
EXPERT
answered 16 days ago
1

Here is an example using the paging api on the cli. In this example there are 25 queues, we are limiting each page to 15.

$ aws sqs list-queues --max-items 15 
{
    "QueueUrls": [
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-01",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-02",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-03",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-04",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-05",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-06",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-07",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-08",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-09",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-10",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-11",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-12",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-13",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-14",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-15"
    ],
    "NextToken": "eyJOZXh0VG9rZW4iOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAxNX0="
}

That is the first 15 queus, when there are more left it will provide a NextToken that you need to include in the next request, like:

$ aws sqs list-queues --max-items 15 --starting-token eyJOZXh0VG9rZW4iOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAxNX0= 
{
    "QueueUrls": [
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-16",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-17",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-18",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-19",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-20",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-21",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-22",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-23",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-24",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-25"
    ]
}

In the second request, we asked for 15 but there were only 10 left so we don't get a NextToken and that tells us we are done.

profile picture
answered 16 days ago
  • Thanks Dave, ya that's what I've been trying, but the problem is, if the total is more than 1000, then it will stop at #1000, and doesn't return any more even with max-items and NextToken paging...

0
Accepted Answer

Ah, this one was tricky. I had to create 1005 queues to figure it out. The 1000 limit appears to be related to the max number of results that will fit in a request made by the client(aws cli itself). When working with more than 1000 queues it looks like you need to adjust both --max-items and --page-size to get through. The --page-size will allow the client to make more requests behind the scenes. Here is how I got through it, your results may vary.

$ aws sqs list-queues --max-items 1000 --page-size 1000
{
    "QueueUrls": [
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-0001",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-0002",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-0003",
...
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-0998",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-0999",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1000"
    ],
    "NextToken": "...TQT09In0="
}

I had to use the --max-items to get the NextToken to show up...

Then to get the additional queues:

$aws sqs list-queues --max-items 1000 --page-size 1000 --starting-token "...TQT09In0="
{
    "QueueUrls": [
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1001",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1002",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1003",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1004",
        "https://sqs.us-west-2.amazonaws.com/123456789123/dk-test-q-1005"
    ]
}
profile picture
answered 14 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions