Will Lambda deployed from image pull the ECR image on every startup

0

Because of Lambda’s ZIP deployment size limitation I’m changing the way I deploy my Lambda functions to use container image.

The deployed function is located within a VPC, so I have a NAT gateway within that VPC to allow public internet traffic, so I need to know if I need to add a VPC endpoints for the ECR services or not (to minimise the traffics going through the NAT gateway).

I’m not sure how/when the lambda pull the image from ECR. is it a 1-time pull that occur when deploying a new image version, or whenever a new lambda instance initialised (cold start)?

In the following post, it says

Lambda also optimizes the image and caches it close to where the functions runs so cold start times are the same as for .zip archives.

But its not clear whether if it’ll still use ECR to get the image or it’ll uses some dedicated location “close to the lambda” ?

3 Answers
1

Please take a look at this document that explains the behavior when a lambda function is deployed as a container image - https://docs.aws.amazon.com/lambda/latest/dg/invocation-images.html

As mentioned in the document, Lambda periodically fetches the associated container image from the Amazon Elastic Container Registry (Amazon ECR) repository. Also, if a function is not invoked for multiple weeks, Lambda reclaims its optimized version, and the function transitions to the Inactive state. To reactivate the function, you must invoke it. Lambda rejects the first invocation and the function enters the Pending state until Lambda re-optimizes the image. The function then returns to the Active state.

So those are the two scenarios in which lambda will pull the container image from ECR.

profile pictureAWS
EXPERT
answered a year ago
  • Thanks for the clarification, that was very useful, I've tested it, once a new image get deployed, the lambda will pull the image only once. Later newly started instances (Cold start ones) won't pull the image from ECR, instead it seems that it uses the already pulled version (According to ECR metrics in the CloudWatch)

  • But I'm still not sure if pulling the image is going through the NAT gateway (located in the same as the lambda's VPC) or not.

0

It is definitely not a one time pull, and it will need to pull on every cold start. However, depending on how often the Lambda executes, it may not pull the image on every execution. Consider the following from Operating Lambda: Performance optimization:

The Lambda service retains the execution environment instead of destroying it immediately after execution. The length of the environment’s lifetime is influenced by various factors that aren’t configurable by the developer today. There are also operational factors in the Lambda services that influence the retention time.

Hence it is not really possible to know how long it will be kept around. Unless your function is firing quite often, or if you are doing something to keep it "warm" (by using EventBridge rules as discussed in the article linked above), it will likely be pulling the image every time.

ECR VPC endpoint is probably the best option for you. Alternatively, note that ingress traffic to EC2 is free, so it might make sense to use a NAT instance instead of a NAT gateway if you are optimizing for cost.

profile picture
EXPERT
bwhaley
answered a year ago
0

I don't think an ECR endpoint makes any difference. In my testing:

  • Created a new VPC
  • No Nat Gateways or Nat Instances
  • Deployed ECR endpoints + S3 endpoint
  • Deployed a Lambda in one of the private subnets, to pull an image from ECR

My assumption was, if I deleted the S3 endpoint, then the Lambda should NOT be able to pull in any images since ECR endpoints require the S3 endpoint to be live. Since I removed the NAT Gateway, there should be no fallback to reach out the internet.

It turns out that the Lambda doesn't need ECR endpoints. I don't know why, but I've confirmed this by deleting the S3 endpoint as a test, and I'm still able to pull images just fine.

This same test would fail on an EKS cluster. I assume this means Lambda is already using the internal AWS network to communicate with ECR.

AlexP
answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions