Skip to content

AWS Lambda performance problem with multi-threading

0

I have a AWS Lambda function that calls a child AWS Lambda function 10 times, each invocation in a separate thread. The child Lambda times itself and returns the timing as the only thing in the body of the response. Here is the parent function:

import boto3
import json
import threading
from stats import Timer

client = boto3.client('lambda')

def do_lambda():
    timer = Timer()
    response = client.invoke(
        FunctionName = 'arn:aws:lambda:us-east-1:497857710590:function:child-function',
        InvocationType = 'RequestResponse',
        Payload = json.dumps({'keys': 1}))
    print(f"Thread done after {timer.stop()}ms: {json.load(response['Payload'])}")

def do_work(n_threads):
    threads = []
    for i in range(n_threads):
        thread = threading.Thread(target=do_lambda, args=())
        thread.start()
        threads.append(thread)

    for thread in threads:
        thread.join()
    

def lambda_handler(event, context):
    do_work(10)
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

The output:

Thread done after 569ms: {'statusCode': 200, 'body': '"553ms"'}
Thread done after 575ms: {'statusCode': 200, 'body': '"558ms"'}
Thread done after 571ms: {'statusCode': 200, 'body': '"555ms"'}
Thread done after 575ms: {'statusCode': 200, 'body': '"559ms"'}
Thread done after 591ms: {'statusCode': 200, 'body': '"575ms"'}
Thread done after 597ms: {'statusCode': 200, 'body': '"580ms"'}
Thread done after 598ms: {'statusCode': 200, 'body': '"573ms"'}
Thread done after 622ms: {'statusCode': 200, 'body': '"603ms"'}
Thread done after 631ms: {'statusCode': 200, 'body': '"575ms"'}
Thread done after 2256ms: {'statusCode': 200, 'body': '"614ms"'}

As you see, the last invocation of the child Lambda took 614ms. However, it took more than 2 seconds for the thread to be actually done. I get this result with one thread taking much longer to be joined consistently. Increasing RAM to 10240 MB does not solve this. If I only use 9 threads, all is fine and all threads are joined in about 650ms.

What is the explanation for this performance limitation of multi-threading under AWS Lambda?

UPDATE: After additional experimentation, I have a different understanding of the question and decided to post that separately under a different title that reflects the new understanding.

asked 3 years ago1.6K views

1 Answer
0

There are multiple things at play here. Off the top of my head (so there are probably others):

You're invoking another Lambda function in each thread. This means interacting with the Lambda control plane (which is a shared service); launching another Lambda function (which has scaling implications as well as other control-plane actions such as rate limiting); and there are underlying things like TCP/TLS setup to the Lambda control plane API and authentication too. All of these things take time.

Adding to this is that the Lambda function you're calling could be a function that is being reused - or not. Depends highly on timing which will be non-deterministic.

Side question: I see your other post - why all the interest in threads? What are you trying to achieve here?

Edit: Probably not helpful to this specific question but perhaps of assistance in general, here's a good article to read (or watch) about the underlying Lambda service architecture: https://www.infoq.com/presentations/aws-lambda-arch/

EXPERT

answered 3 years ago

  • I did additional experimentation and observed that the last invocation of the child Lambda is delayed for almost a second after the nine other invocations of the child Lambda have returned their responses. There are no other Lambda function instances running, but it looks like I am hitting some hard limit. What could that limit be? My goal for parallelization is to read 10,000 items of 40 KB each from AWS DynamoDB in under one second. (I don't want to use step functions) Please see the update section in the question.

  • I think it's worth looking at the article linked above.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.