Skip to content

Invoking Specific Lambda Instances?

0

Is it possible to invoke a specific lambda instance?

I have a use case where a lambda, upon invocation, gathers a large amount of data from S3 based on a DATE parameter on the lambda invocation request and then does some compute. This large amount of data is unique based on the DATE value, so if I invoke 3 concurrent lambdas, each with a different DATE value, each instance will gather a unique set of data during initialization.

Send 3 requests in parallel with the following DATE values:

  • 2024-09-23 - handled by lambda instance 1
  • 2024-09-24 - handled by lambda instance 2
  • 2024-09-25 - handled by lambda instance 3

The part Im curious about is if I send 3 more requests in parallel using the same set of DATE values, is it possible to direct these requests to a specific lambda instance? For example one of the lambda instances was originally fed a DATE of 2024-09-23 and gathered a large amount of data for that DATE. If I can send a follow up request with the same DATE to lambda instance 1, there is a massive performance benefit as the lambda does not have to gather the data again because it currently exists in memory on that lambda instance.

I know this makes my lambdas stateful and goes against the grain of what they are intended to be used for but this would enable a huge performance boost for my use case

asked 2 years ago521 views

2 Answers
2
Accepted Answer

Hi,

Based on this page: https://docs.aws.amazon.com/lambda/latest/operatorguide/execution-environment.html

It is impossible to share execution environment across different executions of the same Lambda (for obvious reasons: security, etc.) : see page

When your functions are invoked, the Lambda service runs your code inside an execution 
environment. Lambda scrubs the memory before it is assigned to an execution environment. 
Execution environments are run on hardware virtualized virtual machines (MicroVMs) which 
are dedicated to a single AWS account. Execution environments are never shared across 
functions and MicroVMs are never shared across AWS accounts. 

But, you can achieve what you want: use AWS Elasticache for Redis. Its recently announced serverless version makes it very cheap.

I use it personally in a current project to share such large amounts of data across different Lambda invocations parts of same Step Function. As you say, it's a huge performance boost.

See https://aws.amazon.com/blogs/aws/amazon-elasticache-serverless-for-redis-and-memcached-now-generally-available/ for details

Best,

Didier

EXPERT

answered 2 years ago

EXPERT

reviewed 2 years ago

EXPERT

reviewed 2 years ago

  • Thank you for the quick response. This is great - I didnt realize Elasticache would work with Lambda. This will help speed things up a bunch

    Cheers!

  • Glad that you like this solution! Thanks for accepting my answer

2

The above mentioned approaches are good. I have a few more recommendations.

Here are a few approaches that can help:

1. Amazon ECS with Fargate:

If you need finer control over instance reuse and in-memory state, consider using Amazon ECS (Elastic Container Service) with Fargate. ECS tasks can maintain a longer-lived state than Lambda, allowing you to keep data in ephemeral storage during the lifecycle of the container across multiple requests. This way, once the data is downloaded for a given DATE, subsequent requests can reuse the temp storage data without the need to fetch it from S3 again. This solution offers more control over how requests are routed and how data is persisted across invocations.

2. Amazon Elastic File System (EFS) with Lambda or ECS:

Lambda now supports integration with Amazon EFS (Elastic File System), which allows you to share a persistent file system across multiple Lambda instances. You could download the data once and store it in EFS, making it accessible to all Lambda instances without needing to fetch it repeatedly from S3. The same approach can be applied in ECS with Fargate, where tasks can be read from EFS to avoid redundant data downloads.

answered 2 years ago

EXPERT

reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.