Skip to content

502 response from cold Lambda with Function URL and docker image and EFS

0

Has anyone experienced 502 errors with Lambda Function URLs when calling container-based functions that haven't been invoked recently?

Our setup:

  • Lambda functions using container images from ECR
  • Function URLs configured with AuthType: NONE
  • Container images (~41MB) stored in ECR
  • Functions in VPC for EFS access

Functions work fine when warm but return 502 if they haven't been invoked for a long time/ever. Response body is

{"Type":"User","message":"The server encountered an error and could not complete your request"}

Response comes back in under 100ms No invocation logs created in CloudWatch, it looks like the function is never reached. After waiting around 10 seconds and retrying, it works fine. Retries before that fail with the same error.

It happens consistently when calling functions that haven't run for a while, and we've seen it both in eu-central-1 and eu-north-1.

Is anyone else seeing this pattern? Especially with Function URLs + container images? Any workarounds you've found besides retrying? Provisioned concurrency isn't feasible for my use case

3 Answers
1
Accepted Answer

What you're experiencing is a common issue with cold starts for Lambda functions using container images, especially when combined with EFS and VPC configurations.

When a Lambda function hasn't been invoked for a while (or ever), AWS needs to initialize the container environment, download the container image, and establish the necessary connections to your VPC and EFS. This initialization process takes time, and if it exceeds certain internal timeouts, you'll receive a 502 error before your function code even executes. This explains why you don't see any CloudWatch logs - the function isn't actually being invoked yet.

The quick response time (under 100ms) suggests that the error is happening during the initialization phase rather than during function execution. The error message "The server encountered an error and could not complete your request" is a generic error that Lambda returns when there's an issue with the function invocation.

Several factors contribute to this behavior:

  1. Container images take longer to initialize than zip-based functions
  2. VPC connectivity adds additional initialization time
  3. EFS mounting requires additional setup time
  4. The size of your container image (41MB) also impacts cold start time

Workarounds to consider:

  1. Implement retry logic in your client with exponential backoff
  2. Use a "warming" strategy where you periodically invoke your functions to keep them warm
  3. Consider splitting your function into two parts - a lightweight non-VPC function that handles initial requests quickly, and a VPC-connected function for operations requiring EFS
  4. Optimize your container image size further if possible
  5. If feasible for some of your functions, consider removing the VPC requirement for functions that don't absolutely need EFS

While provisioned concurrency would be the most direct solution, since you mentioned it's not feasible for your use case, implementing client-side retry logic is likely your best option. The good news is that once the function is warm, it performs reliably as you've observed.
Sources
Public Function URL returning a 403 Forbidden | AWS re:Post
Handling Lambda errors with an API Gateway API - AWS Lambda
Lambda - An error occurred and the request could not be processed. | AWS re:Post

answered 9 months ago

EXPERT

reviewed 9 months ago

  • I looked more into the docs and while it doesn't explicitly say it will return http status 502 or that error message, the docs does indeed say a function will be put into "Inactive" state after a long time without invocations and enter "Pending" state if called again. Calls while in pending will fail.

    https://docs.aws.amazon.com/lambda/latest/dg/functions-states.html

    We'll stick with the retry logic with exponential backoff we've already implemented.

0

If you are calling the Lambda consistently, then you can also use provisioned concurrency for Lambda. This will effectively keep the Lambda warm. [https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html]

EXPERT

answered 9 months ago

  • Thanks, but as I said provisioned concurrency is not feasible for our use case. We have a large number of lambdas where some of them are called very seldom.

0

The work around you can do is to implement automatic retry

import time
import requests

def invoke_with_retry(url, max_retries=3, initial_delay=10):
    for attempt in range(max_retries):
        response = requests.post(url)
        if response.status_code == 502:
            if attempt < max_retries - 1:
                time.sleep(initial_delay)
                continue
        return response
    return response

or Pre-warming with EventBridge Scheduler
Schedule periodic invocations (every 10-15 minutes) to keep functions warm:

{
  "schedule": "rate(10 minutes)",
  "target": "lambda:MyFunction",
  "input": "{\"warmup\": true}"
}
EXPERT

answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.