Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Will Lambda deployed from image pull the ECR image on every startup

0

Because of Lambda’s ZIP deployment size limitation I’m changing the way I deploy my Lambda functions to use container image.

The deployed function is located within a VPC, so I have a NAT gateway within that VPC to allow public internet traffic, so I need to know if I need to add a VPC endpoints for the ECR services or not (to minimise the traffics going through the NAT gateway).

I’m not sure how/when the lambda pull the image from ECR. is it a 1-time pull that occur when deploying a new image version, or whenever a new lambda instance initialised (cold start)?

In the following post, it says

Lambda also optimizes the image and caches it close to where the functions runs so cold start times are the same as for .zip archives.

But its not clear whether if it’ll still use ECR to get the image or it’ll uses some dedicated location “close to the lambda” ?

Topics
ServerlessDevOpsComputeContainers
Tags
DevOpsAmazon Elastic Container Registry (ECR)ContainersAWS Lambda
Language
English
asked 4 years ago4.6K views
3 Answers
1

Please take a look at this document that explains the behavior when a lambda function is deployed as a container image - https://docs.aws.amazon.com/lambda/latest/dg/invocation-images.html

As mentioned in the document, Lambda periodically fetches the associated container image from the Amazon Elastic Container Registry (Amazon ECR) repository. Also, if a function is not invoked for multiple weeks, Lambda reclaims its optimized version, and the function transitions to the Inactive state. To reactivate the function, you must invoke it. Lambda rejects the first invocation and the function enters the Pending state until Lambda re-optimizes the image. The function then returns to the Active state.

So those are the two scenarios in which lambda will pull the container image from ECR.

AWS
EXPERT
answered 4 years ago
  • amersaw
    4 years ago

    Thanks for the clarification, that was very useful, I've tested it, once a new image get deployed, the lambda will pull the image only once. Later newly started instances (Cold start ones) won't pull the image from ECR, instead it seems that it uses the already pulled version (According to ECR metrics in the CloudWatch)

  • amersaw
    4 years ago

    But I'm still not sure if pulling the image is going through the NAT gateway (located in the same as the lambda's VPC) or not.

1

I don't think an ECR endpoint makes any difference. In my testing:

  • Created a new VPC
  • No Nat Gateways or Nat Instances
  • Deployed ECR endpoints + S3 endpoint
  • Deployed a Lambda in one of the private subnets, to pull an image from ECR

My assumption was, if I deleted the S3 endpoint, then the Lambda should NOT be able to pull in any images since ECR endpoints require the S3 endpoint to be live. Since I removed the NAT Gateway, there should be no fallback to reach out the internet.

It turns out that the Lambda doesn't need ECR endpoints. I don't know why, but I've confirmed this by deleting the S3 endpoint as a test, and I'm still able to pull images just fine.

This same test would fail on an EKS cluster. I assume this means Lambda is already using the internal AWS network to communicate with ECR.

answered 3 years ago
  • Rich Andrews
    a year ago

    I've made the exact same observation. It appears that when a lambda function using a container image is created, the lambda function is readily able to access the container image despite the lambda being deployed into a subnet that has no route anywhere other than local. Since it is documented that lambda is able to hold onto an image across cold/warm lambda invocations, I replicated this by deleting and recreating the stack and the new lambda function spun right up with a cold start involving a container from ECR. My interpretation is that indeed container based Lambda deployments do not require Private Link Endpoints nor a route to the internet for that matter.

0

It is definitely not a one time pull, and it will need to pull on every cold start. However, depending on how often the Lambda executes, it may not pull the image on every execution. Consider the following from Operating Lambda: Performance optimization:

The Lambda service retains the execution environment instead of destroying it immediately after execution. The length of the environment’s lifetime is influenced by various factors that aren’t configurable by the developer today. There are also operational factors in the Lambda services that influence the retention time.

Hence it is not really possible to know how long it will be kept around. Unless your function is firing quite often, or if you are doing something to keep it "warm" (by using EventBridge rules as discussed in the article linked above), it will likely be pulling the image every time.

ECR VPC endpoint is probably the best option for you. Alternatively, note that ingress traffic to EC2 is free, so it might make sense to use a NAT instance instead of a NAT gateway if you are optimizing for cost.

EXPERT
answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content