What resource is my Lambda function exhausting?

Question

The following AWS Lambda function uses multi-threading to read 10,000 small objects of about 40KB each from AWS DynamoDB.

```python
import boto3
import threading
import math
import json

# For reading and writing NumPy arrays of np.byte type
class DynamoDB:
    def __init__(self, table_name):
        self.client = boto3.client('dynamodb', region_name="us-east-1")
        self.table_name = table_name
    
    # Batch-read
    def read(self, keys):
        batch_keys = {
            self.table_name: {'Keys': [{'content_id': {'S': id}} for id in keys]}}
        return self.client.batch_get_item(RequestItems=batch_keys)

dynamodb = DynamoDB("mytable")
    
def read_vectors(keys):
    batch_size = 100
    n_batches = math.ceil(len(keys)/batch_size)
    for i in range(n_batches):
        my_keys = keys[i * batch_size : (i + 1) * batch_size]
        dynamodb.read(my_keys)

def concurrent_reads(keys, n_threads):
    threads = []
    keys_per_thread = math.ceil(len(keys)/n_threads)
    for i in range(n_threads):
        my_keys = keys[i * keys_per_thread : (i + 1) * keys_per_thread]
        thread = threading.Thread(target=read_vectors, args=(my_keys,))
        thread.start()
        threads.append(thread)

for thread in threads:
        thread.join()
    
def lambda_handler(event, context):
    keys = [f"vectors/{content_id}" for content_id in range(10000)]
    concurrent_reads(keys, 5) # 5 threads
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }
```

Here are the time measurements:

```
Threads      Time, sec
1            10.5
2             7.1
3             5.5
4             5.2
5             5.4
```

I get similar performance with 2048 MB of RAM and with 10240 MB of RAM. Given that all objects add up to 400 MB and Lambda runs in the same region as DynamoDB, I am definitely not saturating the network bandwidth (especially with 10240 MB, which should give ~25 GiB/s).

What resource is my Lambda exhausting that prevents it from scaling?

**EDIT** I use on-demand capacity and eventually consistent reads.

P.S. The question at SO: https://stackoverflow.com/q/77432589/2725810

Answer

The thing that occurs to me is that each thread might be creating a new TLS connection to DynamoDB. That takes a little bit of time to set up (TCP, TLS, authentication) so you might try playing with thread numbers and batch sizes.

Answer

There is no correlation between amount of memory and network bandwidth. All functions get the same bandwidth. It may seem like smaller functions get less bandwidth, but that is just because it does not have enough CPU to actually drive the max bandwidth.

Saying that, I am not sure if the network bandwidth is your limiting factor.

Answer

Hi,

Since you mention the upper of 10,240 MB, I understand that you know that there is a correlation between allocated memory and available CPU power:  see https://docs.aws.amazon.com/lambda/latest/operatorguide/computing-power.html

```
The amount of memory also determines the amount of virtual CPU available to a function. 
Adding more memory proportionally increases the amount of CPU, increasing the overall 
computational power available. If a function is CPU-, network- or memory-bound, then changing 
the memory setting can dramatically improve its performance.
```

But, as you mention similar perfs at 2,048 MB, I would look at DynamoDb RCUs:  see https://aws.amazon.com/dynamodb/pricing/provisioned/

```
Read capacity unit (RCU): Each API call to read data from your table is a read request. Read requests 
can be strongly consistent, eventually consistent, or transactional. For items up to 4 KB in size, one
 RCU can perform one strongly consistent read request per second. Items larger than 4 KB require additional 
RCUs. For items up to 4 KB in size, one RCU can perform two eventually consistent read requests per second. 
Transactional read requests require two RCUs to perform one read per second for items up to 4 KB. 
For example, a strongly consistent read of an 8 KB item would require two RCUs, an eventually consistent 
read of an 8 KB item would require one RCU, and a transactional read of an 8 KB item would require four RCUs. 
See Read Consistency for more details.
```
You should try a high provisioned capacity in terms of RCU and see if you improve

Finally, you should investigate your perfs under strongly consistent and eventually consistent reads, and get rid of strongly consistent rids if this level of consistency is not needed (or guaranted by another mean in your app). See https://medium.com/expedia-group-tech/dynamodb-guidelines-for-faster-reads-and-writes-3b172b4c2120

Best.

Didier

What resource is my Lambda function exhausting?

관련 콘텐츠