Hi there,
as per the docs here, I have successfully set up everything to invoke a Lambda function from within RDS.
In doing so, I, of course, want to return data, so am using the synchronous mode.
In experimenting with Lambda settings as well as RDS Postgres instance size (t3.micro and m5.large) it seems no normal performance parameters have any impact on the execution time as the extension is executing the Lambda sequentially per row and maxes out at approx. 900 invocations per minute.
Max connections tweaking does not do anything, Lambda concurrency is practically not used and Lambda Memory has no impact. This is the only Lambda active in my account globally, so no limit to concurrency up to the 1k default.
In my scenario, I would want to process tens of thousands of Postgres rows as fast as possible and under three minutes. Right now I'm looking at 50k items with an average of 14 reqs/s resulting in just under 1 hour - both for t3.micro as well as m5.large and an overprovisioned Lambda.
Is there any way on how to achieve this using this functionality other than flipping the ETL and making it pull based vs. push based? Basically massively increase Lambda invocations from the RDS side.
EDIT: It just so happens that the - unchangeable - limit for GetPolicy API reqs/s is at 15 for Lambda.
Is each invocation via the RDS extension invoking this API endpoint?
