Migrate python scripts from Local to AWS

0

Hey,

I have a local python script that extracts data using a google API and stores data in a csv file.

  1. I want to replicate this process on AWS by storing the csv data in an S3 bucket.
  2. Later, I want this file to run daily and update the same csv file in S3 How should I approach this requirement?

I am new to AWS and would appreciate your help!

2개 답변
0

Create a lambda service role, that would have access to your s3 bucket and the location, where you would store csv file in that bucket. Here is how to create lambda execution role, you'll have to add additional policy to this role based on what resources you need to access from lambda so the lambda execution role.

Create a lambda function with python runtime(python3.10 or so)and adjust your python script in that lambda function and import all the required boto3 modules. Make sure that you use the role while creating this function which you'd have set up in first step. Here is how to create lambda function from console. Within your code, you may already be using pandas dataframe to get the data from google API, you may need to add pandas module or any required modules as layer to your lambda function. Here is how to add layers to lambda functions.

Create an eventbridge rule in cloudwatch event and make a cron schedule with target as this lambda function. Here is the tutorial of Schedule AWS Lambda functions using EventBridge.

Hope you find this this information helpful.

Comment here if you have additional questions, happy to help.

Abhishek

profile pictureAWS
전문가
답변함 8달 전
0

Abhishek's answer is spot on. If you want help to automate the creation of the Lambda, you can use AWS SAM to automate the deployment of the Lambda, as well as creating the Eventbridge rule to schedule the function. E.g. with SAM CLI, then 5. Scheduled Task, and this is the piece in the template.yaml that will run the function:

# This example runs every hour.
      Events:
        CloudWatchEvent:
          Type: Schedule
          Properties:
            Schedule: cron(0 * * * ? *)
profile picture
답변함 8달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠