Lambda csv file to S3 is blank

Question

I am writing a Lambda function using Python. My Lambda function is querying my test db and storing data in a list of dict. File loads into S3 but content is the name of the file and not the data from the list of dict that are returned by the query.
I need the rows from the list of dicts along with the keys as headers loaded into the csv file.

What am I doing wrong?

Steps Taken:
1. Granted IAM role S3 Full Access
2. Created VPC S3 Endpoint
3. Created S3 bucket access point

Below is my code section that generates the csv and uploads to S3

```
try:
    conn = pymysql.connect(host=rds_proxy_host, user=username, passwd=password, db=db_name, connect_timeout=10)
except pymysql.MySQLError as e:
    logger.error("ERROR: Unexpected error: Could not connect to MySQL instance.")
    logger.error(e)
    sys.exit(1)
logger.info("SUCCESS: Connection to RDS for MySQL instance succeeded")

def lambda_handler(event, context):
    with conn.cursor(pymysql.cursors.DictCursor) as cur:
        cur.execute("SELECT customerNumber,orderNumber,DATE_FORMAT(orderDate, '%Y-%m-%d') orderDate,orderLineNumber,productCode,quantityOrdered,cast(priceEach as char) priceEach,cast(cost as char) cost FROM classicmodels.v_orderinfo where quantityOrdered < 100 limit 10;")
        rowcounts = cur.rowcount
        if rowcounts > 1:
            result = cur.fetchall()
            fieldnames = list(result[0].keys())
            with open('/tmp/csv_file.csv','w',encoding='utf8', newline='') as csvfile:
                dict_writer = csv.DictWriter(csvfile, fieldnames = fieldnames)
                dict_writer.writeheader()
                dict_writer.writerows(result)
            s3client.put_object(Bucket = s3bucketname, Body = '/tmp/csv_file.csv',Key = 'csv_file.csv')    
            conn.commit
            for row in result:
                logger.info(row)
            snsclient.publish(TopicArn=snsarn,Message='MySQLConnectTestSNS Demo has successfully executed and rows returned.')
            print("Message published")
            return print('Added {count} for MySQL table'.format(count=str(rowcounts)))
    
```

Accepted Answer

Your code is saying that the body (the contents of file file to upload) is the string `/tmp/csv_file.csv`. The `Body` parameter requires "bytes or a seekable file-like object" as per [the documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/put_object.html).

Perhaps you're looking for the [upload_file method](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/upload_file.html)?

Another way would be to use the `StringIO` library in Python to build the file in memory, then pass that to `put_object` as the `Body`. Depends on how much memory the function has; and you may also want to see how much space `/tmp` has too depending on the size of the query result from the database.

Answer

Hello.

Looking at this, it looks like put_object needs to specify the data rather than the file name to be uploaded.  
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/put_object.html

So, I thought that if I want to upload the file as is, I need to write the code as follows.  
```
with open('/tmp/csv_file.csv', 'rb') as csvfile:
    csv_content = csvfile.read()
s3client.put_object(Bucket=s3bucketname, Body=csv_content, Key='csv_file.csv') 
```
Alternatively, I think it is possible to use upload_file() and do the following.  
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/upload_file.html
```
s3client.upload_file(Filename='/tmp/csv_file.csv', Bucket=s3bucketname, Key='csv_file.csv')
```

Lambda csv file to S3 is blank

相關內容