Trigger a Glue Job using Lambda for a Specific S3 file

0

I want to trigger 2 different Glue jobs based on the file that arrives in S3 bucket. The names of these files are different and based on it, the corresponding Glue job should be triggered using a Lambda function. Below is the code I have written. But when i place the file in S3, the triggering is not happening. But when i test this manually, the jobs are triggered as expected. What can be the possible reason for the Automate process?? I have placed the policy & IAM roles as expected. Still do i need to check any particular role or any problem in my code?? Please help!!

import json import boto3

def lambda_handler(event, context): glue = boto3.client(service_name='glue') print("The lambda handler is invoked")

if 'Records' in event:
    s3_record = event['Records'][0]['s3']
    print("IF clause has been invoked")

    bucket = s3_record['bucket']['name']
    object_key = s3_record['object']['key']
    print(f"bucket details are: {bucket}")
    print(f"key details are: {key}")
    
    try:
        if object_key.endswith(".csv") and "file_name_a" in object_key:
            # Trigger the second Glue job if the condition is met
            response = glue.start_job_run(JobName='my_job_1')
            print(f"Started Glue job run for file_name_b: {response}")
        if object_key.endswith(".csv") and "file_name_b" in object_key:
            # Trigger the second Glue job if the condition is met
            response = glue.start_job_run(JobName='my_job_2')
            print(f"Started Glue job run for file_name_b: {response}")
    except Exception as e:
        print(e)
return {
    'statusCode': 200,
    'body': json.dumps('Lambda function executed successfully!')
}
Joe
preguntada hace 7 meses517 visualizaciones
1 Respuesta
0

Hello.

Does this mean that Lambda does not work even after uploading the file to S3?
Have you made the settings described in the following document?
Also, could you share how your S3 event trigger is configured?
https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

profile picture
EXPERTO
respondido hace 7 meses
  • Thanks for the comments Riku. The setting were given as expected. My issue was, I didn't initiate a boto client for S3 at the beginning and this caused the issue. I should have used "s3 = boto3.client('s3')" This piece should have been written after the "import boto3". After doing this, the code worked smoothly in AUTO process.

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas