Getting "Rate exceeded" for AWS::Lambda::LayerVersion resource deployment via SAM

Question

I have a sam template with a common runtime dependency defined in `Resources` as:

```
RuntimeDependenciesLayer:
    Type: AWS::Serverless::LayerVersion
    Metadata:
      BuildMethod: makefile
    Properties:
      Description: Runtime dependencies for Lambdas
      LayerName: !Sub ${AppName}-${Environment}-runtime-dependencies
      ContentUri: ./
      CompatibleRuntimes:
        - nodejs18.x
      RetentionPolicy: Retain
```

This dependency is shared across all the lambda functions using `Globals`, like this:
```
Globals:
  Function:
    Runtime: nodejs18.x
    Tracing: Active
    Layers:
      - !Ref RuntimeDependenciesLayer
    ...
```

I use a Makefile for the dependency layer:
```
.PHONY: build-RuntimeDependenciesLayer build-lambda-common
.PHONY: build-pingFn

build-pingFn:
	$(MAKE) HANDLER=src/handlers/ping.ts build-lambda-common

BASE_DIR = $(ARTIFACTS_DIR)/../../..
REPO_ROOT = $(BASE_DIR)/../..
PACKAGE_PATH = backend/workflow

build-lambda-common:
	cd $(BASE_DIR) && pnpm install --frozen-lockfile --filter="." --filter="...{./$(PACKAGE_PATH)}..."
	cd $(BASE_DIR) && rm -rf dist
	cd $(BASE_DIR) && echo "{\"extends\": \"./tsconfig.json\", \"include\": [\"${HANDLER}\"] }" > tsconfig-only-handler.json
	cd $(BASE_DIR) && pnpm build --build tsconfig-only-handler.json
	cp -r $(BASE_DIR)/dist "$(ARTIFACTS_DIR)/"

build-RuntimeDependenciesLayer:
	mkdir tmp
	# Copy the repo content to a temporary folder, excluding the .git folder and node_modules folders
	rsync -a --exclude ".git" --exclude "*node_modules*" $(REPO_ROOT) tmp
	# Install dependencies in that temp folder
	cd tmp && pnpm install --frozen-lockfile --shamefully-hoist --ignore-scripts --prod --filter="." --filter="...{./$(PACKAGE_PATH)}..."
	mkdir -p "$(ARTIFACTS_DIR)/nodejs/node_modules"
	# Copy resulting node_nodules (the L flag is to follow links and get real copies)
	cp -rL tmp/node_modules/* "$(ARTIFACTS_DIR)/nodejs/node_modules"
	cp -rL tmp/$(PACKAGE_PATH)/node_modules/* "$(ARTIFACTS_DIR)/nodejs/node_modules"
```

The issue is, when I deploy multiple clients (multiple sam deployments) in parallel (via pipeline), the `AWS::Lambda::LayerVersion` resource in Cloudformation template fails to create for some of the clients, with the following error:
```
"Rate exceeded (Service: AWSLambdaInternal; Status Code: 400; Error Code: ThrottlingException; Request ID: a1b2-c3d4-e5f6; Proxy: null)" (RequestToken: g7h8-i9j0-k1l2, HandlerErrorCode: GeneralServiceException)
```

Is there a way to make sure that the throttling doesn't happen, as a subset of deployments fail because of this issue?
Any advice/suggestion is appreciated.

Thanks!

re:Post Agent · Accepted Answer

The issue you're experiencing with rate exceeded errors for AWS::Lambda::LayerVersion resource deployments via SAM is related to AWS service limits and throttling. When deploying multiple clients in parallel, you're likely hitting the API rate limits for creating Lambda layers.

To mitigate this issue and reduce the likelihood of throttling errors, you can consider the following approaches:

1. Implement exponential backoff and retry logic in your deployment process. This can help handle temporary throttling issues by automatically retrying failed operations with increasing delays between attempts.

2. Stagger your deployments instead of running them all in parallel. You could introduce a slight delay between each client deployment to spread out the API requests over time.

3. Reduce the frequency of layer updates. If possible, update your layers less often or only when necessary, rather than with every deployment.

4. Use a shared layer across multiple clients instead of creating separate layers for each client. This can significantly reduce the number of layer creation operations.

5. Consider using AWS Lambda container images instead of layers for larger dependencies. Container images have a higher size limit (10GB) and may be more suitable for your use case, especially if you have many dependencies.

6. If your layer doesn't change frequently, you could implement a check to see if the layer already exists before attempting to create a new version. This would reduce unnecessary API calls.

7. Reach out to AWS support to request a quota increase for Lambda layer operations if you consistently need to perform a high volume of layer creations.

By implementing one or more of these strategies, you should be able to reduce the occurrence of throttling errors and improve the reliability of your parallel deployments.  
**Sources**  
[Lambda Package Exceeds 60MB: Solutions for Large Dependencies? | AWS re:Post](https://repost.aws/questions/QUoXSNWrkdSf61jVqG7IwBxg/lambda-package-exceeds-60mb-solutions-for-large-dependencies)  
[AWS::Serverless::LayerVersion - AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-layerversion.html)  
[Using AWS SAM with layers - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/layers-sam.html)

Getting "Rate exceeded" for AWS::Lambda::LayerVersion resource deployment via SAM

Add your answer

Relevant content