DynamoDB first request is very slow

0

Hi,
During building a simple serverless CRUD API using API Gateway, Lambda(Java 8) and DynamoDB. I came across a problem related to a slow response to the first DynamoDB request. Generally, responses are in the 10s of milliseconds however when I'm executing my lambda function for the first time I am seeing response time around 7s.
In my lambda function I'm using Java SDK v2 I also used all the practices from this post: [https://aws.amazon.com/blogs/developer/tuning-the-aws-java-sdk-2-x-to-reduce-startup-time]
I did some tests and here are the results of the Lambda execution with DynamoDB request:

|| Avg Response Time | | --- | --- | --- | |Lambda 512mb| 7s | |Lambda 1024mb| 5s | |Lambda 2048mb| 3.5s | |Creating only DynamoDB Client 512mb Lambda | 3s | Note: All of these responses are from cold start Lambda. The avg time for the cold start of my Java Lambdas is 1.5s.

Are there any official resources that are saying that the first DynamoDB request will be so slow?

Thanks

3 Answers
3
Accepted Answer

It's worthwhile using something like X-Ray to determine where the delay is coming from. It's probable that the issue is related to the Java SDK but without actual data it's hard to say. I do a lot of work in Lambda using Python - the cold start times are very low and the time-to-first request with DynamoDB is similarly very short.

The response time is lower for Lambda functions with more memory because the CPU allocated to them is proportional to the memory - more memory equals more CPU so that explains some of the difference there.

In general I recommend to customers that they optimise their code to reduce the cold start time. One way of doing that is lazy-loading/initialising libraries only when they are needed rather than at function start.

profile pictureAWS
EXPERT
answered 2 years ago
profile picture
EXPERT
reviewed 2 days ago
3

While I agree with most of what Brettski mentions, from a DynamoDB perspective you will have to understand how DynamoDB makes use of extensive metadata caching. For your initial request, you will have to create a TCP connection, as DynamoDB uses HTTP endpoints. Furthermore, you will have to do Authorization, Authentication and other metadata checks related to your tables partitions and items locations. This metadata is cached and so for subsequent requests you will notice a great reduction in latency and this is normal while interacting with DynamoDB.

I do not agree with lazy loading resources in a Lambda function, as creating clients inside a Lambda handler will cause you to create the client for every invocation, and will introduce increased latencies. Ensure to create your client outside of the Lambda handler so it is shared across invocations.

profile pictureAWS
EXPERT
answered 2 years ago
  • Good point about creating the DynamoDB client connection outside the handler!

  • Yeah, I'm creating all resources outside the handler. Thank you for explaining what's going on underneath.

  • Bear in mind that creating all resources outside the handler will increase the cold start time. So be judicious in what you do there.

2

Important thing to note here is that this problem is significantly more acute during development. You're making updates to your code, deploying new versions, and as a result getting cold-starts much more frequently comparing to the production environment, where your functions run unchanged for longer period of times.

To add to the answers above, there are several techniques you can use to speed up cold-starts in your first function invocation. My answer is based on the assumption that you need to invoke DynamoDB (DDB) from your handler code.

  1. Raising memory allocation. Lambda allocates CPU to your function based on the memory allocation, so the more memory you allocate to your function the more CPU power it will have. With more CPU power your first request to DynamoDB (and to practically any other service) via the AWS SDK will be faster. This is a very simple step that usually results in major cold-start times improvements. Note that it MIGHT result in more cost since Lambda pricing model is based on execution time and allocated memory size.

  2. Digging a little bit deeper into the possible cause. The AWS SDK has lazily loaded components, which means some of the SDK internals will be fully initialized only once you make your first request (e.g. DDB GetItem). Subsequent requests will be faster because the client is fully initialized. During function initialization Lambda allocates more CPU resources, so your function init will happen faster. Combining these two facts, what you can do is to add a simple DDB request in your init block, right after you initialize the function. It can be a simple request to get a dummy item from DDB, but it will be enough to fully "prime" the client during the init phase, which has more CPU resources allocated. This approach will marginally raise your cold-start init time, but also significantly lower your cold-start invoke time. This approach can be very impactful under low memory condition (<=1gb), it will be less effective as you allocate more memory to your function, since as a result it will be getting more CPU as well.

    See a sample code snippet below just to illustrate the approach.

    public class App implements RequestHandler<Object, String> {
        static DynamoDbClient ddbClient;
        static {
            ddbClient = DynamoDbClient.builder().build();
            getAnyDummyItemFromDynamoDB(); // This will happen during function init
        }
        @Override
        public String handleRequest(Object input, Context context) {
            System.out.println("> function handler1");
            doSomeRealDynamoDBOperations(); // This will happen during function handler invoke
            return null;
        }
    }
    
  3. Enabling Tiered Compilation for your Java Lambda's. This blog gives a great overview of how this can be done.

  4. Once you've applied the above optimization techniques, you can use the AWS Lambda Power Tuning tool. It can help you to identify the best memory configuration based on your execution time/cost requirements.

AWS
answered 2 years ago
  • Wow! The Tiered Compilation speeded it up to ~4.7 seconds. Thank you!

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions