How to ensure using the latest lambda layer version when deploying with CloudFormation and SAM?

Question

Hi, we use CloudFormation and SAM to deploy our Lambda (Node.js) functions. All our Lambda functions has a layer set through `Globals`. When we make breaking changes in the layer code we get errors during deployment because new Lambda functions are rolled out to production with old layer and after a few seconds *(~40 seconds in our case)* it starts using the new layer. For example, let's say we add a new class to the layer and we import it in the function code then we get an error that says `NewClass is not found` for a few seconds during deployment *(this happens because new function code still uses old layer which doesn't have `NewClass`)*.

Is it possible to ensure new lambda function is always rolled out with the latest layer version?

Example CloudFormation template.yaml:
```
Globals:
  Function:
    Runtime: nodejs14.x
    Layers:
      - !Ref CoreLayer

Resources:
  CoreLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: core-layer
      ContentUri: packages/coreLayer/dist
      CompatibleRuntimes:
        - nodejs14.x
    Metadata:
      BuildMethod: nodejs14.x

ExampleFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: example-function
      CodeUri: packages/exampleFunction/dist
```

SAM build: `sam build --base-dir . --template ./template.yaml`

SAM package: `sam package --s3-bucket example-lambda --output-template-file ./cf.yaml`

Example CloudFormation deployment events, as you can see new layer (`CoreLayer123abc456`) is created before updating the Lambda function so it should be available to use in the new function code but for some reasons Lambda is updated and deployed with the old layer version for a few seconds:

| Timestamp | Logical ID | Status | Status reason |
| --- | --- | --- | --- |
2022-05-23 16:26:54 | stack-name | UPDATE_COMPLETE | -
2022-05-23 16:26:54 | CoreLayer789def456 | DELETE_SKIPPED | -
2022-05-23 16:26:53 | v3uat-farthing | UPDATE_COMPLETE_CLEANUP_IN_PROGRESS | -
2022-05-23 16:26:44 | ExampleFunction | UPDATE_COMPLETE | -
2022-05-23 16:25:58 | ExampleFunction | UPDATE_IN_PROGRESS | -
2022-05-23 16:25:53 | CoreLayer123abc456 | CREATE_COMPLETE | -
2022-05-23 16:25:53 | CoreLayer123abc456 | CREATE_IN_PROGRESS | Resource creation Initiated
2022-05-23 16:25:50 | CoreLayer123abc456 | CREATE_IN_PROGRESS | -
2022-05-23 16:25:41 | stack-name | UPDATE_IN_PROGRESS | User Initiated

Accepted Answer

After further investigation, it is not possible to to apply all changes in a single step. However, there is a way around it. Instead of invoking the LATEST version of you function, publish a new version after all the changes are done and invoke the specific version. You can do this also using an ALIAS.

Answer

After a couple of tests it turns out that the issue is caused by the order of the changes from changeset. Looks like changes are applied one by one. For example for the following changeset it updates the old function code while still using the old layer and then updates the function layer with the latest version because `Layers` change item comes after `Code` change item.

```
{
  "resourceChange":{
    "logicalResourceId":"ExampleFunction",
    "action":"Modify",
    "physicalResourceId":"example-function",
    "resourceType":"AWS::Lambda::Function",
    "replacement":"False",
    "moduleInfo":null,
    "details":[
      {
        "target":{
          "name":"Layers",
          "requiresRecreation":"Never",
          "attribute":"Properties"
        },
        "causingEntity":null,
        "evaluation":"Dynamic",
        "changeSource":"DirectModification"
      },
      {
        "target":{
          "name":"Code",
          "requiresRecreation":"Never",
          "attribute":"Properties"
        },
        "causingEntity":null,
        "evaluation":"Static",
        "changeSource":"DirectModification"
      },
      {
        "target":{
          "name":"Layers",
          "requiresRecreation":"Never",
          "attribute":"Properties"
        },
        "causingEntity":"CoreLayer123abc456",
        "evaluation":"Static",
        "changeSource":"ResourceReference"
      }
    ],
    "changeSetId":null,
    "scope":[
      "Properties"
    ]
  },
  "hookInvocationCount":null,
  "type":"Resource"
}
```
But in some deployments the order is the other way around, such as:

```
{
  "resourceChange":{
    ...
    "details":[
      ...
      {
        "target":{
          "name":"Layers",
          "requiresRecreation":"Never",
          "attribute":"Properties"
        },
        "causingEntity":"CoreLayer123abc456",
        "evaluation":"Static",
        "changeSource":"ResourceReference"
      },
      {
        "target":{
          "name":"Code",
          "requiresRecreation":"Never",
          "attribute":"Properties"
        },
        "causingEntity":null,
        "evaluation":"Static",
        "changeSource":"DirectModification"
      }
    ],
    ...
}
```
In this case it updates the old function with the latest layer version and then updates the function code with the updated one. So for a couple of seconds old code is invoked with the latest layer version.

So does it possible to apply all these changes in only one single step? Similar to [Atomicity in databases](https://en.wikipedia.org/wiki/Atomicity_(database_systems))

Answer

This is strange. Layers is just a way to package code that is deployed with the function. Once a function is created or updated, it includes the layer code and it doesn't change until the function is updated again. This means that if a function took an old version of the layer, it should fail all the time. While a function is being deployed, some of the invocations will go to the old version of the function, but that version also includes the old version of the layer. Once the deployment is done the invocations will go to the new version, which includes new layer version.

Saying the above, I would recommend that each function includes its own dependencies directly and not use layers for that. This way your functions' code will be smaller as each will only include the dependencies it requires. Also, if you change something in one function that needs a change in the dependencies, it will not affect the other functions using the same layer.

How to ensure using the latest lambda layer version when deploying with CloudFormation and SAM?

Relevant questions

AWS SAM CLI: Deploy AWS Lambda with least privilege access

SAM & Docker Builds-- How do they update (not working for me)?

Update Existing Lambda Function using SAM.

Lambda Functions raise forbidden error until stack deleted and redeployed

When to use AWS CLI versus AWS SAM CLI for Lambda function deployment

HttpApi timeout setting missing from SAM cloudformation template

How to ensure using the latest lambda layer version when deploying with CloudFormation and SAM?

SAM deploy does not deploy Layer dependencies to S3

Deploying Lambda functions from GitHub "serverlessly" with our Code* services

How to create a lambda layer for a CodeStar project?

Timestamp	Logical ID	Status	Status reason
2022-05-23 16:26:54	stack-name	UPDATE_COMPLETE	-
2022-05-23 16:26:54	CoreLayer789def456	DELETE_SKIPPED	-
2022-05-23 16:26:53	v3uat-farthing	UPDATE_COMPLETE_CLEANUP_IN_PROGRESS	-
2022-05-23 16:26:44	ExampleFunction	UPDATE_COMPLETE	-
2022-05-23 16:25:58	ExampleFunction	UPDATE_IN_PROGRESS	-
2022-05-23 16:25:53	CoreLayer123abc456	CREATE_COMPLETE	-
2022-05-23 16:25:53	CoreLayer123abc456	CREATE_IN_PROGRESS	Resource creation Initiated
2022-05-23 16:25:50	CoreLayer123abc456	CREATE_IN_PROGRESS	-
2022-05-23 16:25:41	stack-name	UPDATE_IN_PROGRESS	User Initiated