S3 client get_object hangs


I wrote a Lambda function to process ingested files in S3 in Python 3.11. The function hangs on get_object:

print(f"Got to get_object('{bucket}', '{key}')")
s3object = s3.get_object(Bucket=bucket, Key=key)
print(f"which line hangs? If you see this it is read.")

The log shows the first line, then Task timed out after 10.05 seconds (it has a 10s timeout set).

I know the object can be loaded quickly because I wrote another lambda which processes the same file in less than a second.

Considering this logically, there's probably something different between the 2 lambdas that causes this issue, but I'm stumped. One of them reads the file, transforms it, and writes the output to another location - it works fine. This one connects to RedShift and would (if it could load the file), write the data to RedShift. It hangs.

Both lambdas receive events that a new file exists and they both have the same permissions on the bucket:

          - Action:
                - s3:Copy*
                - s3:Delete*
                - s3:Get*
                - s3:List*
                - s3:Put*
              Effect: Allow
                - arn:aws:s3:::my-ingest-bucket
                - arn:aws:s3:::my-ingest-bucket/*

The lambda hangs whether the notification is due to a ObjectCreated:Copy (when the first lambda moves the source file) or ObjectCreated:Put (when i drop the file in S3 to manually test).

Any suggestions how to troubleshoot this?

preguntada hace 4 meses96 visualizaciones
1 Respuesta

To me, this signals that the lambda that is hanging may not have the correct execution role.

respondido hace 3 meses
  • Thanks for the suggestion.
    It turned out the cause of the issue was networking: I had attached the lambda to a private VPC, which hid S3. To test it I just removed the lambda function from the VPC, and this made S3 accessible (but lost access to RedShift).

    What I was missing was an egress rule in each security group with a prefix referring to the S3 VPC endpoint (my post doesn't mention it, but I had created a VPC endpoint for S3 access already).

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas