By using AWS re:Post, you agree to the Terms of Use
/How to ensure using the latest lambda layer version when deploying with CloudFormation and SAM?/

How to ensure using the latest lambda layer version when deploying with CloudFormation and SAM?

0

Hi, we use CloudFormation and SAM to deploy our Lambda (Node.js) functions. All our Lambda functions has a layer set through Globals. When we make breaking changes in the layer code we get errors during deployment because new Lambda functions are rolled out to production with old layer and after a few seconds (~40 seconds in our case) it starts using the new layer. For example, let's say we add a new class to the layer and we import it in the function code then we get an error that says NewClass is not found for a few seconds during deployment (this happens because new function code still uses old layer which doesn't have NewClass).

Is it possible to ensure new lambda function is always rolled out with the latest layer version?

Example CloudFormation template.yaml:

Globals:
  Function:
    Runtime: nodejs14.x
    Layers:
      - !Ref CoreLayer

Resources:
  CoreLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: core-layer
      ContentUri: packages/coreLayer/dist
      CompatibleRuntimes:
        - nodejs14.x
    Metadata:
      BuildMethod: nodejs14.x

  ExampleFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: example-function
      CodeUri: packages/exampleFunction/dist

SAM build: sam build --base-dir . --template ./template.yaml

SAM package: sam package --s3-bucket example-lambda --output-template-file ./cf.yaml

Example CloudFormation deployment events, as you can see new layer (CoreLayer123abc456) is created before updating the Lambda function so it should be available to use in the new function code but for some reasons Lambda is updated and deployed with the old layer version for a few seconds:

TimestampLogical IDStatusStatus reason
2022-05-23 16:26:54stack-nameUPDATE_COMPLETE-
2022-05-23 16:26:54CoreLayer789def456DELETE_SKIPPED-
2022-05-23 16:26:53v3uat-farthingUPDATE_COMPLETE_CLEANUP_IN_PROGRESS-
2022-05-23 16:26:44ExampleFunctionUPDATE_COMPLETE-
2022-05-23 16:25:58ExampleFunctionUPDATE_IN_PROGRESS-
2022-05-23 16:25:53CoreLayer123abc456CREATE_COMPLETE-
2022-05-23 16:25:53CoreLayer123abc456CREATE_IN_PROGRESSResource creation Initiated
2022-05-23 16:25:50CoreLayer123abc456CREATE_IN_PROGRESS-
2022-05-23 16:25:41stack-nameUPDATE_IN_PROGRESSUser Initiated
3 Answers
1
Accepted Answer

After further investigation, it is not possible to to apply all changes in a single step. However, there is a way around it. Instead of invoking the LATEST version of you function, publish a new version after all the changes are done and invoke the specific version. You can do this also using an ALIAS.

EXPERT
answered a month ago
  • Many thanks @Uri, lambda versions worked! I added a lambda version:

    ExampleFunctionVersion:
      Type: AWS::Lambda::Version
      DeletionPolicy: Delete
      Properties:
        FunctionName: !Ref ExampleFunction
    

    and replaced all function references with !Ref ExampleFunctionVersion.

    I also tried AutoPublishAlias: live, it worked too but I had to add an AWS::Lambda::Version resource to be able to set DeletionPolicy: Delete

0

This is strange. Layers is just a way to package code that is deployed with the function. Once a function is created or updated, it includes the layer code and it doesn't change until the function is updated again. This means that if a function took an old version of the layer, it should fail all the time. While a function is being deployed, some of the invocations will go to the old version of the function, but that version also includes the old version of the layer. Once the deployment is done the invocations will go to the new version, which includes new layer version.

Saying the above, I would recommend that each function includes its own dependencies directly and not use layers for that. This way your functions' code will be smaller as each will only include the dependencies it requires. Also, if you change something in one function that needs a change in the dependencies, it will not affect the other functions using the same layer.

EXPERT
answered a month ago
  • Thanks for the quick reply!

    While a function is being deployed, some of the invocations will go to the old version of the function, but that version also includes the old version of the layer. Once the deployment is done the invocations will go to the new version, which includes new layer version.

    This is exactly what we expect but for some reasons, during the deployment new functions are invoked with the old layer!

    I would recommend that each function includes its own dependencies directly and not use layers for that.

    We use layers to share the common codes where we introduce breaking changes inevitable but this shouldn't be an issue if new functions use new layer and old functions use old layer.

0

After a couple of tests it turns out that the issue is caused by the order of the changes from changeset. Looks like changes are applied one by one. For example for the following changeset it updates the old function code while still using the old layer and then updates the function layer with the latest version because Layers change item comes after Code change item.

{
  "resourceChange":{
    "logicalResourceId":"ExampleFunction",
    "action":"Modify",
    "physicalResourceId":"example-function",
    "resourceType":"AWS::Lambda::Function",
    "replacement":"False",
    "moduleInfo":null,
    "details":[
      {
        "target":{
          "name":"Layers",
          "requiresRecreation":"Never",
          "attribute":"Properties"
        },
        "causingEntity":null,
        "evaluation":"Dynamic",
        "changeSource":"DirectModification"
      },
      {
        "target":{
          "name":"Code",
          "requiresRecreation":"Never",
          "attribute":"Properties"
        },
        "causingEntity":null,
        "evaluation":"Static",
        "changeSource":"DirectModification"
      },
      {
        "target":{
          "name":"Layers",
          "requiresRecreation":"Never",
          "attribute":"Properties"
        },
        "causingEntity":"CoreLayer123abc456",
        "evaluation":"Static",
        "changeSource":"ResourceReference"
      }
    ],
    "changeSetId":null,
    "scope":[
      "Properties"
    ]
  },
  "hookInvocationCount":null,
  "type":"Resource"
}

But in some deployments the order is the other way around, such as:

{
  "resourceChange":{
    ...
    "details":[
      ...
      {
        "target":{
          "name":"Layers",
          "requiresRecreation":"Never",
          "attribute":"Properties"
        },
        "causingEntity":"CoreLayer123abc456",
        "evaluation":"Static",
        "changeSource":"ResourceReference"
      },
      {
        "target":{
          "name":"Code",
          "requiresRecreation":"Never",
          "attribute":"Properties"
        },
        "causingEntity":null,
        "evaluation":"Static",
        "changeSource":"DirectModification"
      }
    ],
    ...
}

In this case it updates the old function with the latest layer version and then updates the function code with the updated one. So for a couple of seconds old code is invoked with the latest layer version.

So does it possible to apply all these changes in only one single step? Similar to Atomicity in databases

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions