Bug when calling Lambda from Postgres RDS. Works for short function runs < 5 seconds, but not if the call lasts 10 seconds

1

I have found a bug when calling Lambda from Postgres RDS. I am using Postgres version 13. I have created a one minute video nailing down the bug: https://youtu.be/Y1ALUxEwzGk

I am using the following query to call the lambda: SELECT payload->>'answer' from aws_lambda.invoke(aws_commons.create_lambda_function_arn('arn:aws:lambda:eu-central-1:xxxxxx:function:ChatGPT', 'eu-central-1'), '{"prompt": ""}'::json)

Here is my working python code:

import time

def lambda_handler(event, context):
    time.sleep(1)
    return {
        'statusCode': 200,
        "answer": "Test"
    }

If I change this line:

time.sleep(1)

to

time.sleep(10)

Then the call will fail.

Postgres version: PostgreSQL 13.10 on aarch64-unknown-linux-gnu, compiled by gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-6), 64-bit

Also, the timeout of the lambda is 3 minutes (far more than 10 seconds): https://drive.google.com/file/d/1iMEpZDM3PCA9b6glJnAh93tspW0V3BvD/view?usp=drivesdk

A related issue is that there seems to be a hard timeout of the aws_lambda.invoke of 100 seconds, and I would really like that to be adjustable. My use case is that I am querying chatgpt from inside postgres and it can really take a long time to return an answer, sometimes.

The error message after 103 seconds is: ERROR: invoke API failed DETAIL: AWS Lambda client returned 'Unable to connect to endpoint'. CONTEXT: SQL function "invoke" statement 1

So no timeout error message.

  • Definitely not a lambda timeout issue.

  • Is there any error message? What is it?

  • Sorry for my slow reply. The error message after 103 seconds is: ERROR: invoke API failed DETAIL: AWS Lambda client returned 'Unable to connect to endpoint'. CONTEXT: SQL function "invoke" statement 1

    So no timeout error message. The video in my original post really gives much more context on this.

    Also, I am very sure that this is an actual bug. Is there any way to notify the developers of the postgres extension?

  • Raised an issue with the service team.

3 Answers
1
Accepted Answer

It seems there is a currently undocumented parameter that defines the timeout.

set aws_lambda.request_timeout_ms in your parameter group to the appropriate value and give it a try.

profile pictureAWS
EXPERT
Uri
answered 8 months ago
  • Did you have a chance to test this parameter? Did it work for you?

  • Thank you for following up. There is definitely a bug here since the query should return after around 10 seconds in the original example, but instead ran for 100 seconds. I tried to set this before running the query: set aws_lambda.request_timeout_ms = 600000;

    And ran this query again: SELECT payload->>'answer' from aws_lambda.invoke(aws_commons.create_lambda_function_arn('arn:aws:lambda:eu-central-1:xxxxxx:function:ChatGPT', 'eu-central-1'), '{"prompt": ""}'::json)

    I got the same result as before (after 103 seconds), but with an additional line:

    WARNING: parameter "aws_lambda.request_timeout_ms" cannot be changed now

  • You need to set it in the Parameter Group that is used by your RDS cluster.

  • Thank you Uri! I will need to look into this once the bug has been fixed. The bug will get fixed, right?

  • Currently the only bug is the lack of documentation for these parameters. Please try setting them in your cluster's parameter group and try again.

0

This may or may not be the cause, however calling lambda functions isn’t supported on 13.10

Invoking an AWS Lambda function is supported in these RDS for PostgreSQL versions:

  1. All PostgreSQL 15 versions
  2. PostgreSQL 14.1 and higher minor versions
  3. PostgreSQL 13.2 and higher minor versions
  4. PostgreSQL 12.6 and higher minor versions
profile picture
EXPERT
answered 8 months ago
0

Hi, you can adjust timeout on Lambda execution up to minutes, which is beyond what you need-

See https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html#configuration-timeout-console on how to adjust it for your use case.

All Lambda quotas are defined here: https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

Best,

Didier

profile pictureAWS
EXPERT
answered 8 months ago
  • Thank you for looking into my problem! Setting the timeout on the Lambda is, though not the issue here. As you can see I have already adjusted the timeout enough. There is a problem in the communication between RDS and Lambda.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions