How to merge 2 csv files from S3 bucket using Lambda.

Question

Hi, I'm new to AWS. I have been trying to create a Lambda function that gets triggered every time a new file gets uploaded to an S3 bucket which, for the time being, will only ever contain 2 files. The function should join the 2 files. I'm using Python for this and have created a layer to be able to use pandas. My current attempt is posted below and yields and error message about the name of the bucket being wrong.

Any help would be much appreciated!

def lambda_handler(event, context):
    
    bucket = 'arn:aws:s3:::name-of-the-bucket/subfolder/'
    filename_1 = 'first_file.csv'
    filename_2 = 'second_file.csv'
    
    s3 = boto3.client('s3')
    
    first_obj = s3.get_object(Bucket= bucket, Key= filename_1)
    second_obj = s3.get_object(Bucket= bucket, Key= filename_2)
    
    first_df = pd.read_csv(first_obj['Body'])
    second_df = pd.read_csv(second_obj['Body'])
    
    joined_df = first_df.merge(second_df, on=['ID'], how='left')
    
    return joined_df

Accepted Answer

The bucket name should be the name of bucket without subfolder.

Change your code to

```
bucket = 'name-of-the-bucket'
filename_1 = 'subfolder/first_file.csv'
filename_2 = 'subfolder/second_file.csv'
```

--Syd

How to merge 2 csv files from S3 bucket using Lambda.

Relevanter Inhalt