Trigger a Glue Job using Lambda for a Specific S3 file

0

I want to trigger 2 different Glue jobs based on the file that arrives in S3 bucket. The names of these files are different and based on it, the corresponding Glue job should be triggered using a Lambda function. Below is the code I have written. But when i place the file in S3, the triggering is not happening. But when i test this manually, the jobs are triggered as expected. What can be the possible reason for the Automate process?? I have placed the policy & IAM roles as expected. Still do i need to check any particular role or any problem in my code?? Please help!!

import json import boto3

def lambda_handler(event, context): glue = boto3.client(service_name='glue') print("The lambda handler is invoked")

if 'Records' in event:
    s3_record = event['Records'][0]['s3']
    print("IF clause has been invoked")

    bucket = s3_record['bucket']['name']
    object_key = s3_record['object']['key']
    print(f"bucket details are: {bucket}")
    print(f"key details are: {key}")
    
    try:
        if object_key.endswith(".csv") and "file_name_a" in object_key:
            # Trigger the second Glue job if the condition is met
            response = glue.start_job_run(JobName='my_job_1')
            print(f"Started Glue job run for file_name_b: {response}")
        if object_key.endswith(".csv") and "file_name_b" in object_key:
            # Trigger the second Glue job if the condition is met
            response = glue.start_job_run(JobName='my_job_2')
            print(f"Started Glue job run for file_name_b: {response}")
    except Exception as e:
        print(e)
return {
    'statusCode': 200,
    'body': json.dumps('Lambda function executed successfully!')
}
Joe
質問済み 7ヶ月前517ビュー
1回答
0

Hello.

Does this mean that Lambda does not work even after uploading the file to S3?
Have you made the settings described in the following document?
Also, could you share how your S3 event trigger is configured?
https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

profile picture
エキスパート
回答済み 7ヶ月前
  • Thanks for the comments Riku. The setting were given as expected. My issue was, I didn't initiate a boto client for S3 at the beginning and this caused the issue. I should have used "s3 = boto3.client('s3')" This piece should have been written after the "import boto3". After doing this, the code worked smoothly in AUTO process.

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ