Trigger a Glue Job using Lambda for a Specific S3 file

0

I want to trigger 2 different Glue jobs based on the file that arrives in S3 bucket. The names of these files are different and based on it, the corresponding Glue job should be triggered using a Lambda function. Below is the code I have written. But when i place the file in S3, the triggering is not happening. But when i test this manually, the jobs are triggered as expected. What can be the possible reason for the Automate process?? I have placed the policy & IAM roles as expected. Still do i need to check any particular role or any problem in my code?? Please help!!

import json import boto3

def lambda_handler(event, context): glue = boto3.client(service_name='glue') print("The lambda handler is invoked")

if 'Records' in event:
    s3_record = event['Records'][0]['s3']
    print("IF clause has been invoked")

    bucket = s3_record['bucket']['name']
    object_key = s3_record['object']['key']
    print(f"bucket details are: {bucket}")
    print(f"key details are: {key}")
    
    try:
        if object_key.endswith(".csv") and "file_name_a" in object_key:
            # Trigger the second Glue job if the condition is met
            response = glue.start_job_run(JobName='my_job_1')
            print(f"Started Glue job run for file_name_b: {response}")
        if object_key.endswith(".csv") and "file_name_b" in object_key:
            # Trigger the second Glue job if the condition is met
            response = glue.start_job_run(JobName='my_job_2')
            print(f"Started Glue job run for file_name_b: {response}")
    except Exception as e:
        print(e)
return {
    'statusCode': 200,
    'body': json.dumps('Lambda function executed successfully!')
}
Joe
已提问 7 个月前517 查看次数
1 回答
0

Hello.

Does this mean that Lambda does not work even after uploading the file to S3?
Have you made the settings described in the following document?
Also, could you share how your S3 event trigger is configured?
https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

profile picture
专家
已回答 7 个月前
  • Thanks for the comments Riku. The setting were given as expected. My issue was, I didn't initiate a boto client for S3 at the beginning and this caused the issue. I should have used "s3 = boto3.client('s3')" This piece should have been written after the "import boto3". After doing this, the code worked smoothly in AUTO process.

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则