Trigger a Glue Job using Lambda for a Specific S3 file

0

I want to trigger 2 different Glue jobs based on the file that arrives in S3 bucket. The names of these files are different and based on it, the corresponding Glue job should be triggered using a Lambda function. Below is the code I have written. But when i place the file in S3, the triggering is not happening. But when i test this manually, the jobs are triggered as expected. What can be the possible reason for the Automate process?? I have placed the policy & IAM roles as expected. Still do i need to check any particular role or any problem in my code?? Please help!!

import json import boto3

def lambda_handler(event, context): glue = boto3.client(service_name='glue') print("The lambda handler is invoked")

if 'Records' in event:
    s3_record = event['Records'][0]['s3']
    print("IF clause has been invoked")

    bucket = s3_record['bucket']['name']
    object_key = s3_record['object']['key']
    print(f"bucket details are: {bucket}")
    print(f"key details are: {key}")
    
    try:
        if object_key.endswith(".csv") and "file_name_a" in object_key:
            # Trigger the second Glue job if the condition is met
            response = glue.start_job_run(JobName='my_job_1')
            print(f"Started Glue job run for file_name_b: {response}")
        if object_key.endswith(".csv") and "file_name_b" in object_key:
            # Trigger the second Glue job if the condition is met
            response = glue.start_job_run(JobName='my_job_2')
            print(f"Started Glue job run for file_name_b: {response}")
    except Exception as e:
        print(e)
return {
    'statusCode': 200,
    'body': json.dumps('Lambda function executed successfully!')
}
Joe
demandé il y a 7 mois518 vues
1 réponse
0

Hello.

Does this mean that Lambda does not work even after uploading the file to S3?
Have you made the settings described in the following document?
Also, could you share how your S3 event trigger is configured?
https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

profile picture
EXPERT
répondu il y a 7 mois
  • Thanks for the comments Riku. The setting were given as expected. My issue was, I didn't initiate a boto client for S3 at the beginning and this caused the issue. I should have used "s3 = boto3.client('s3')" This piece should have been written after the "import boto3". After doing this, the code worked smoothly in AUTO process.

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions