error while reading csv from S3 using pandas

0

Hi Team, I am getting an error "Install s3fs to access S3" while trying to access a file from S3 in AWS lambda. However I tried importing all possible packages but still I am not able to read the file.

  • import json import boto3 import pandas as pd

    def lambda_handler(event, context):

    df1 = pd.read_csv('s3://himanrcctest/size_and_fit/Logs_2023-07-15.csv001',delimiter="\t")
    return str(df1
    

    I have already run below commands: pip3 install s3fs pip3 install fs_s3fs however I got this error while running above commnd - awscli 1.29.60 requires botocore==1.31.60, but you have botocore 1.31.17 which is incompatible. boto3 1.28.60 requires botocore<1.32.0,>=1.31.60, but you have botocore 1.31.17 which is incompatible.

asked 7 months ago284 views
1 Answer
0

Hello.

What if you run the command below to update botocore to the latest version?
https://docs.aws.amazon.com/efs/latest/ug/install-botocore.html

sudo pip3 install botocore --upgrade

Also, since neither pandas nor s3fs are included in the Lambda package, you need to create a layer using the steps in the document below.

mkdir python
pip3 install pandas -t ./python/
pip3 install s3fs -t ./python/
zip -r lambda_layer.zip python/
aws lambda publish-layer-version --layer-name "lambda_layer_name" --zip-file fileb://lambda_layer.zip --compatible-runtimes python3.11
profile picture
EXPERT
answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions