Migrate python scripts from Local to AWS

0

Hey,

I have a local python script that extracts data using a google API and stores data in a csv file.

  1. I want to replicate this process on AWS by storing the csv data in an S3 bucket.
  2. Later, I want this file to run daily and update the same csv file in S3 How should I approach this requirement?

I am new to AWS and would appreciate your help!

質問済み 8ヶ月前355ビュー
2回答
0

Create a lambda service role, that would have access to your s3 bucket and the location, where you would store csv file in that bucket. Here is how to create lambda execution role, you'll have to add additional policy to this role based on what resources you need to access from lambda so the lambda execution role.

Create a lambda function with python runtime(python3.10 or so)and adjust your python script in that lambda function and import all the required boto3 modules. Make sure that you use the role while creating this function which you'd have set up in first step. Here is how to create lambda function from console. Within your code, you may already be using pandas dataframe to get the data from google API, you may need to add pandas module or any required modules as layer to your lambda function. Here is how to add layers to lambda functions.

Create an eventbridge rule in cloudwatch event and make a cron schedule with target as this lambda function. Here is the tutorial of Schedule AWS Lambda functions using EventBridge.

Hope you find this this information helpful.

Comment here if you have additional questions, happy to help.

Abhishek

profile pictureAWS
エキスパート
回答済み 8ヶ月前
0

Abhishek's answer is spot on. If you want help to automate the creation of the Lambda, you can use AWS SAM to automate the deployment of the Lambda, as well as creating the Eventbridge rule to schedule the function. E.g. with SAM CLI, then 5. Scheduled Task, and this is the piece in the template.yaml that will run the function:

# This example runs every hour.
      Events:
        CloudWatchEvent:
          Type: Schedule
          Properties:
            Schedule: cron(0 * * * ? *)
profile picture
回答済み 8ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ