Retrieve list of buckets from all regions using boto3

Question

I was trying to retrieve the list of buckets from all regions using boto3, however I am unable to list the buckets from the correct region

So far I have tried `location['LocationConstraint']`, which comes up as `None`
I have also tried the following as well but it didn't work.

Any help is appreciated, thank you

```
if client.head_bucket(Bucket=bucket['Name'])['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region'] == 'us-east-1':
                print ("bucketname %s " % s3_bucket.name)
```

**Code**
```
import json
import boto3

bucketlist = []

def lambda_handler(event, context):

# Get list of regions
    ec2 = boto3.client('ec2')
    regions = ec2.describe_regions().get('Regions',[] )

# Iterate over regions
    for region in regions:
    
        print ("*************** Checking region  --   %s " % region['RegionName'])
        reg=region['RegionName']
    
        client = boto3.client('s3', region_name=reg)

response = client.list_buckets()
        for bucket in response['Buckets']:
            s3 = boto3.resource('s3', region_name=reg)
            s3_bucket = s3.Bucket(bucket['Name'])
            if client.head_bucket(Bucket=bucket['Name'])['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region'] == 'us-east-1':
                print ("bucketname %s " % s3_bucket.name)
                bucketlist.append(s3_bucket)
    return {
        "statusCode": 200
    }
```

**Output:** The buckets `testbucket1` and `testbucket2` are only available in `us-east-1` region
```
START RequestId: e22f6ac0-7bb9-4e2b-84d7-5512ce97acfa Version: $LATEST
*************** Checking region  --   eu-north-1 
bucketname testbucket1 
bucketname testbucket2 
*************** Checking region  --   ap-south-1 
bucketname testbucket1 
bucketname testbucket2 
[...]
*************** Checking region  --   us-east-1 
bucketname testbucket1
bucketname testbucket2
```

**Expected Output:**
```
START RequestId: e22f6ac0-7bb9-4e2b-84d7-5512ce97acfa Version: $LATEST
*************** Checking region  --   eu-north-1 
*************** Checking region  --   ap-south-1 
*************** Checking region  --   eu-west-3 
*************** Checking region  --   eu-west-2 
*************** Checking region  --   eu-west-1 
*************** Checking region  --   ap-northeast-3 
*************** Checking region  --   ap-northeast-2 
*************** Checking region  --   ap-northeast-1 
*************** Checking region  --   sa-east-1 
*************** Checking region  --   ca-central-1 
*************** Checking region  --   ap-southeast-1 
*************** Checking region  --   ap-southeast-2 
*************** Checking region  --   eu-central-1 
*************** Checking region  --   us-east-1 
bucketname testbucket1 
bucketname testbucket2 
*************** Checking region  --   us-east-2 
*************** Checking region  --   us-west-1 
*************** Checking region  --   us-west-2 
```

Accepted Answer

The `head_bucket` function automatically redirects to the correct region in the background, so the `x-amz-bucket-region` header will always end up being the bucket's region no matter which region you called it from originally.

Changing the condition to be the following should give you the output you want:
```
if client.head_bucket(Bucket=bucket['Name'])['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region'] == reg:
```

However, iterating over every region like that is making a lot of unnecessary requests, so it might be worth looking at the [`get_bucket_location`](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.get_bucket_location) function which returns the `LocationConstraint` value, which - as you've found -  is `None` for buckets in us-east-1. Buckets in any other region should return the actual region name. Just iterating over all buckets once and calling `get_bucket_location` on each of them should speed up the response.

Retrieve list of buckets from all regions using boto3

相关内容