Bug when calling Lambda from Postgres RDS. Works for short function runs < 5 seconds, but not if the call lasts 10 seconds

1

I have found a bug when calling Lambda from Postgres RDS. I am using Postgres version 13. I have created a one minute video nailing down the bug: https://youtu.be/Y1ALUxEwzGk

I am using the following query to call the lambda: SELECT payload->>'answer' from aws_lambda.invoke(aws_commons.create_lambda_function_arn('arn:aws:lambda:eu-central-1:xxxxxx:function:ChatGPT', 'eu-central-1'), '{"prompt": ""}'::json)

Here is my working python code:

import time

def lambda_handler(event, context):
    time.sleep(1)
    return {
        'statusCode': 200,
        "answer": "Test"
    }

If I change this line:

time.sleep(1)

to

time.sleep(10)

Then the call will fail.

Postgres version: PostgreSQL 13.10 on aarch64-unknown-linux-gnu, compiled by gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-6), 64-bit

Also, the timeout of the lambda is 3 minutes (far more than 10 seconds): https://drive.google.com/file/d/1iMEpZDM3PCA9b6glJnAh93tspW0V3BvD/view?usp=drivesdk

A related issue is that there seems to be a hard timeout of the aws_lambda.invoke of 100 seconds, and I would really like that to be adjustable. My use case is that I am querying chatgpt from inside postgres and it can really take a long time to return an answer, sometimes.

The error message after 103 seconds is: ERROR: invoke API failed DETAIL: AWS Lambda client returned 'Unable to connect to endpoint'. CONTEXT: SQL function "invoke" statement 1

So no timeout error message.

  • Definitely not a lambda timeout issue.

  • Is there any error message? What is it?

  • Sorry for my slow reply. The error message after 103 seconds is: ERROR: invoke API failed DETAIL: AWS Lambda client returned 'Unable to connect to endpoint'. CONTEXT: SQL function "invoke" statement 1

    So no timeout error message. The video in my original post really gives much more context on this.

    Also, I am very sure that this is an actual bug. Is there any way to notify the developers of the postgres extension?

  • Raised an issue with the service team.

3 Antworten
1
Akzeptierte Antwort

It seems there is a currently undocumented parameter that defines the timeout.

set aws_lambda.request_timeout_ms in your parameter group to the appropriate value and give it a try.

profile pictureAWS
EXPERTE
Uri
beantwortet vor 8 Monaten
  • Did you have a chance to test this parameter? Did it work for you?

  • Thank you for following up. There is definitely a bug here since the query should return after around 10 seconds in the original example, but instead ran for 100 seconds. I tried to set this before running the query: set aws_lambda.request_timeout_ms = 600000;

    And ran this query again: SELECT payload->>'answer' from aws_lambda.invoke(aws_commons.create_lambda_function_arn('arn:aws:lambda:eu-central-1:xxxxxx:function:ChatGPT', 'eu-central-1'), '{"prompt": ""}'::json)

    I got the same result as before (after 103 seconds), but with an additional line:

    WARNING: parameter "aws_lambda.request_timeout_ms" cannot be changed now

  • You need to set it in the Parameter Group that is used by your RDS cluster.

  • Thank you Uri! I will need to look into this once the bug has been fixed. The bug will get fixed, right?

  • Currently the only bug is the lack of documentation for these parameters. Please try setting them in your cluster's parameter group and try again.

0

This may or may not be the cause, however calling lambda functions isn’t supported on 13.10

Invoking an AWS Lambda function is supported in these RDS for PostgreSQL versions:

  1. All PostgreSQL 15 versions
  2. PostgreSQL 14.1 and higher minor versions
  3. PostgreSQL 13.2 and higher minor versions
  4. PostgreSQL 12.6 and higher minor versions
profile picture
EXPERTE
beantwortet vor 9 Monaten
0

Hi, you can adjust timeout on Lambda execution up to minutes, which is beyond what you need-

See https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html#configuration-timeout-console on how to adjust it for your use case.

All Lambda quotas are defined here: https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

Best,

Didier

profile pictureAWS
EXPERTE
beantwortet vor 9 Monaten
  • Thank you for looking into my problem! Setting the timeout on the Lambda is, though not the issue here. As you can see I have already adjusted the timeout enough. There is a problem in the communication between RDS and Lambda.

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen