Migrate python scripts from Local to AWS

0

Hey,

I have a local python script that extracts data using a google API and stores data in a csv file.

  1. I want to replicate this process on AWS by storing the csv data in an S3 bucket.
  2. Later, I want this file to run daily and update the same csv file in S3 How should I approach this requirement?

I am new to AWS and would appreciate your help!

已提问 8 个月前356 查看次数
2 回答
0

Create a lambda service role, that would have access to your s3 bucket and the location, where you would store csv file in that bucket. Here is how to create lambda execution role, you'll have to add additional policy to this role based on what resources you need to access from lambda so the lambda execution role.

Create a lambda function with python runtime(python3.10 or so)and adjust your python script in that lambda function and import all the required boto3 modules. Make sure that you use the role while creating this function which you'd have set up in first step. Here is how to create lambda function from console. Within your code, you may already be using pandas dataframe to get the data from google API, you may need to add pandas module or any required modules as layer to your lambda function. Here is how to add layers to lambda functions.

Create an eventbridge rule in cloudwatch event and make a cron schedule with target as this lambda function. Here is the tutorial of Schedule AWS Lambda functions using EventBridge.

Hope you find this this information helpful.

Comment here if you have additional questions, happy to help.

Abhishek

profile pictureAWS
专家
已回答 8 个月前
0

Abhishek's answer is spot on. If you want help to automate the creation of the Lambda, you can use AWS SAM to automate the deployment of the Lambda, as well as creating the Eventbridge rule to schedule the function. E.g. with SAM CLI, then 5. Scheduled Task, and this is the piece in the template.yaml that will run the function:

# This example runs every hour.
      Events:
        CloudWatchEvent:
          Type: Schedule
          Properties:
            Schedule: cron(0 * * * ? *)
profile picture
已回答 8 个月前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则