Lambda random long execution while running QLDB query

0

I have a lambda triggered by a SQS FIFO queue when there are messages on this queue. Basically this lambda is getting the message from the queue and connecting to QLDB through a VPC endpoint in order to run a simple SELECT query and a subsequent INSERT query. The table selected by the query has a index for the field used in the where condition.

Flow (all the services are running "inside" a VPC):

SQS -> Lambda -> VPC interface endpoint -> QLDB

Query SELECT: SELECT FIELD1, FIELD2 FROM TABLE1 WHERE FIELD3 = "ABCDE"

Query INSERT: INSERT INTO TABLE1 .....

This lambda is using a shared connection/session on QLDB and this is how I'm connecting to it:

import { QldbDriver, RetryConfig } from 'amazon-qldb-driver-nodejs'

let driverQldb: QldbDriver
const ledgerName = 'MyLedger'

export function connectQLDB(): QldbDriver {
  if ( !driverQldb ) {
    const retryLimit = 4
    const retryConfig = new RetryConfig(retryLimit)
    const maxConcurrentTransactions = 1500
    driverQldb = new QldbDriver(ledgerName, {}, maxConcurrentTransactions, retryConfig)
  }
  return driverQldb
}

When I run a load test that simulates around 200 requests/messages per second to that lambda in a time interval of 15 minutes, I'm starting facing a random long execution for that lambda while running the queries on QLDB (mainly the SELECT query). Sometimes the same query retrieves data around 100ms and sometimes it takes more than 40 seconds which results in lambda timeouts. I have changed lambda timeout to 1 minute but this is not the best approch and sometimes it is not enough too.

The VPC endpoint metrics are showing around 250 active connections and 1000 new connections during this load test execution. Is there any QLDB metric that could help to identify the root cause of this behavior?

Could it be related to some QLDB limitation (like the 1500 active sessions described here: https://docs.aws.amazon.com/qldb/latest/developerguide/limits.html#limits.default) or something related to concurrency read/write iops?

1 Answer
1

Hi @thiagoscodeler,

Please check out the proposed answer on stackoverflow to this question you posted there.

https://stackoverflow.com/questions/73780825/lambda-random-long-execution-while-running-qldb-query/73816697#73816697

AWS
answered 2 years ago
  • Hi @bwinchester-aws, thank you. I'll check the stackoverflow answer.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions