Skip to content

Lambda - An error occurred and the request could not be processed.

0

We have a a lambda service which has start to give us an error message

{
 "Message": "An error occurred and the request could not be processed." , 
"Type": "Service"
}

This happens when we have more than around 200 concurrent executions of the lambda being called. When using lower quantities of concurrent calls to this lambda we don't see this message.

When this message is returned, it doesn't actually run the code on the lambda, it returns this message and nothing else.

The lambda is running rust code on Amazon Linux 2023 through a function url and is connected to EFS. The connected EFS is not saturated and the lambda itself does not have a restriction on the numbers of concurrent executions it is allowed to have.

We used to run many more concurrent executions of this code before ( a couple of months ago) and since Friday I needed to increase our calls to this service once again but keep seeing this message. It seems to be sporadic as to which calls to the lambda return this message. Most of them succeed (return the expected data) but a large percentage of them do not.

I am really confused by this issue and the error message doesn't give any information as to why this may be happening. There are also no clues in the cloudwatch logs.

Any help in trying to figure this out would be helpful.

  • I wanted to follow up on this. The only way I found to solve this issue was by throttling my calls to the lambda function. I found that a delay of 4ms between calls to the lambda function seemed to work well and I was able to test it up to 500 concurrent calls.

    I am only using 200 in production so this is good.

    I'm not really happy with this solution, especially as the routine can often require 100k calls to be completed so obviously time added by this throttling is not ideal.

    I have been a bit surprised by this problem as I had never seen it happening before ( ie. a few months back at least when we last altered the number of concurrent calls) and I've not been able to find any info on AWS about max calls/sec etc to the lambda functions.

    It is working for now but I will probably need to come back and figure out a more intelligent way to throttle ( potentially in batches or growing delay times etc). I'm not really sure.

    If anyone has any ideas on this or can point me to some AWS info regarding this issue then it would be appreciated.

asked a year ago279 views

2 Answers
0

The error message "An error occurred and the request could not be processed" with type "Service" is likely related to capacity issues when your Lambda function is experiencing high concurrency. Since this happens when you exceed approximately 200 concurrent executions, there are several potential causes and solutions to consider:

  1. Insufficient capacity error: AWS might not have enough available compute capacity to execute your function at that moment, especially at higher concurrency levels. This can happen even if your function worked fine previously at higher concurrency.

  2. EFS connection issues: Since your Lambda is connected to EFS, there could be mounting or connection issues when many concurrent executions try to access the file system simultaneously. Even if the EFS itself isn't saturated, the connection process might be experiencing issues.

  3. Concurrency throttling: While you mentioned there's no restriction on concurrency, Lambda might still be throttling requests due to account-level concurrency limits or service quotas.

To address this issue, you could try:

  • Increase the function's memory allocation, which might provide access to different hardware resources
  • Implement retry logic in your client code to handle these temporary capacity issues
  • Set up reserved concurrency for your function to ensure a specific amount is always available
  • Consider distributing your workload across multiple regions
  • Check if your function is timing out during initialization or execution
  • Review CloudWatch metrics for throttling, errors, and invocation metrics
  • Ensure your function code gracefully handles duplicate events and has proper error handling

If the problem persists after trying these solutions, it might be worth contacting AWS Support to investigate if there are any underlying issues specific to your account or the regions you're using.
Sources
Insufficient capacity error when triggering a Lambda function | AWS re:Post
Troubleshoot invocation issues in Lambda - AWS Lambda
How Lambda handles errors and retries with asynchronous invocation - AWS Lambda

answered a year ago

AWS
EXPERT

reviewed a year ago

0

Based on your description, I suspect that is is tied to a concurrency setting/limit in your account, especially when occurring regularly at a very round number like 200. Look deep at those settings and the settings for concurrency of the other Lambda functions in the same account.

AWS
EXPERT

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content