Read AWS Bucket with Python

Question

Greetings,
I am trying to read JSON data in an AWS Bucket using Python.
I don't get any errors, but don't return any data in Python script??
Any suggestions to resolve this. would be appreciated.

```
 ~/.aws/config
import os, sys
import http, httptools, aiohttp
import urllib, urllib3
import boto, boto3
from boto3 import Session
from boto.s3.connection import S3Connection
#from boto import OpenSSL
from boto.s3 import bucket
import glob, os, json
import pandas as pd
import numpy

### SET Directory
json_dir = 'losses-costs/2022_12_23/France/'
### SELECT Files
json_pattern = os.path.join(json_dir, '*.json')
file_list = glob.glob(json_pattern)
### IMPORT files
dfs = []
for file in file_list:
    with open(file) as f:
        json_data = pd.json_normalize(json.loads(f.read()))
        json_data['site'] = file.rsplit("/", 1)[-1]
    print(f)
 #print(df)
```
There is null output, the folder has four files??
Any assistance, appreciated.
Thanks

Answer

When using Boto you need to initialize an S3 client.
Then you use that client to get the object.
You then need to convert the object to a usable format.

Look at this link for samples on how to use Boto3 with S3 buckets. Specifically look at the `Get an object from a bucket` sample.
https://docs.aws.amazon.com/code-library/latest/ug/python_3_s3_code_examples.html

Answer

Hi,
I see you are importing several `boto` related libraries, but you script doesn't actually make use of them as far as I can tell. Based on the script you shared, I believe you need to add the code that lists the objects in your bucket. Once you do that, you can continue to develop your code to load each file into memory, or download it to disk, etc.

Here is a basic example using `boto3` which should list the objects in your bucket.

```
import boto3
session = boto3.Session( aws_access_key_id='', aws_secret_access_key='')

s3 = session.resource('s3')
my_bucket = s3.Bucket('')

for my_bucket_object in my_bucket.objects.all():
    print(my_bucket_object.key)
```

Read AWS Bucket with Python

Relevant content