Trigger a Glue Job using Lambda for a Specific S3 file

0

I want to trigger 2 different Glue jobs based on the file that arrives in S3 bucket. The names of these files are different and based on it, the corresponding Glue job should be triggered using a Lambda function. Below is the code I have written. But when i place the file in S3, the triggering is not happening. But when i test this manually, the jobs are triggered as expected. What can be the possible reason for the Automate process?? I have placed the policy & IAM roles as expected. Still do i need to check any particular role or any problem in my code?? Please help!!

import json import boto3

def lambda_handler(event, context): glue = boto3.client(service_name='glue') print("The lambda handler is invoked")

if 'Records' in event:
    s3_record = event['Records'][0]['s3']
    print("IF clause has been invoked")

    bucket = s3_record['bucket']['name']
    object_key = s3_record['object']['key']
    print(f"bucket details are: {bucket}")
    print(f"key details are: {key}")
    
    try:
        if object_key.endswith(".csv") and "file_name_a" in object_key:
            # Trigger the second Glue job if the condition is met
            response = glue.start_job_run(JobName='my_job_1')
            print(f"Started Glue job run for file_name_b: {response}")
        if object_key.endswith(".csv") and "file_name_b" in object_key:
            # Trigger the second Glue job if the condition is met
            response = glue.start_job_run(JobName='my_job_2')
            print(f"Started Glue job run for file_name_b: {response}")
    except Exception as e:
        print(e)
return {
    'statusCode': 200,
    'body': json.dumps('Lambda function executed successfully!')
}
Joe
已提問 7 個月前檢視次數 518 次
1 個回答
0

Hello.

Does this mean that Lambda does not work even after uploading the file to S3?
Have you made the settings described in the following document?
Also, could you share how your S3 event trigger is configured?
https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

profile picture
專家
已回答 7 個月前
  • Thanks for the comments Riku. The setting were given as expected. My issue was, I didn't initiate a boto client for S3 at the beginning and this caused the issue. I should have used "s3 = boto3.client('s3')" This piece should have been written after the "import boto3". After doing this, the code worked smoothly in AUTO process.

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南