Skip to content

AWS SAM CodeDeploy Lambda deployment failure CodeDeployServiceRole AWSLambda request limit exceeded

0

I have a backend service consisting of 29 Lambda functions deployed with AWS SAM with versioning enabled. In the past few months deployments became flaky and sometimes only succeed after several re-runs, failing with "InternalFailure" error code (visible in CloudFormation, which also now marks it as the root cause of failures). When issues first began, I noticed several "ThrottlingException" errors from CodeDeploy in CloudTrail during deployment, so I experimented with the AWS_MAX_ATTEMPTS and AWS_RETRY_MODE environment variables to allow more retries and also retry smarter; the effects were visible (now the deployment would continue after the default 3 retries), but there were still deployment failures with the same cause. Now I decided to change the CodeDeploy deployment preference from "Linear50PercentEvery1Minute" to "AllAtOnce", hoping it would result in less CodeDeploy calls, which also helped, I no longer see "ThrottlingException" errors in CloudTrail but deployments still fail from time to time. Now, when I check CodeDeploy's deployments, I see the following error message at the time when the CloudFormation deployment fails: The deployment failed because the number of requests by the IAM role arn:aws:iam::[account_id]:role/[stack_name]-CodeDeployServiceRole-[suffix] exceeded the request limit for AWSLambda. Avoid retrying API requests at a high rate. Trying to fix this, I checked the Lambda quotas docs, which says there is a 15 request per second non-increasable quota on Lambda API calls (excluding GetFunction and GetPolicy requests). I checked in CloudTrail, the deployment occasionally goes above 15 calls, mostly by doing GetProvisionedConcurrencyConfig, GetAlias20150331 and UpdateAlias20150331 calls. I'd like to ask if Lambda versioning and CodeDeploy is expected to work with this (and higher) number of Lambda functions, or is this solution only limited to a limited number of functions. If it's supposed to work, how can fix the issue of CodeDeploy's IAM service role getting itself rate limited by calling the AWSLambda API during deployment? As I understand the problematic calls are not done by AWS SAM during deployment, but internally by CodeDeploy, so maybe it can only be resolved by not deploying all of my functions at once, but only a few at a time (or else I need to give up on Lambda versioning)? I checked similar questions, they mention checking the root cause in CloudFormation and the logs in CloudTrail, I've done both, but now it seems to me the issue is clear but there's no simple solution to fix it. Thanks in advance!

1 Answer
0

The issue you're experiencing with AWS SAM CodeDeploy Lambda deployment failures is related to the number of Lambda functions being deployed simultaneously and the resulting API request limits being exceeded. This is a known challenge when deploying a large number of Lambda functions, especially with versioning enabled.

Lambda versioning and CodeDeploy are designed to work with multiple functions, but there are practical limits due to API request quotas. In your case, with 29 Lambda functions and versioning enabled, you're hitting these limits during deployment.

To address this issue, consider the following approaches:

  1. Staggered Deployments: Instead of deploying all 29 functions at once, split them into smaller groups and deploy them in stages. This can help reduce the number of simultaneous API calls to Lambda.

  2. Increase Deployment Duration: If possible, adjust the deployment configuration to spread the updates over a longer period. This can help reduce the API call frequency.

  3. Reduce Versioning Frequency: If feasible, consider reducing the frequency of creating new versions for functions that don't change often.

  4. Use Canary Deployments: Implement canary deployments for critical functions, which can help reduce the number of simultaneous updates.

  5. Optimize Lambda Configuration: Review and optimize your Lambda function configurations to reduce unnecessary API calls during deployment.

  6. AWS Support: If these solutions don't resolve the issue, consider reaching out to AWS Support for guidance on handling deployments with a large number of Lambda functions.

It's important to note that while Lambda versioning and CodeDeploy can work with a high number of functions, there are practical limits due to API quotas. Your approach of splitting deployments or reducing the number of functions updated simultaneously might be necessary for reliable deployments with this many functions.

Remember, the 15 requests per second quota for Lambda API calls is a hard limit and cannot be increased. Therefore, managing your deployment strategy to work within this constraint is crucial for successful large-scale Lambda deployments.

Sources
Error codes for AWS CodeDeploy - AWS CodeDeploy
Error while deploying the HTTP API + Lambda using AWS SAM | AWS re:Post

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.