Limitation API Gateway/Lambda with querying


Hey all! Hope your are doing well. I have been trying to write a query service for some internal databases in my VPC. My current setup is API Gateway with a Lambda that queries the database which works fine, but unfortunately I ran into two issues:

  • API Gateway default timeout is 30s which is not very long for queries.

  • Lambda response size limit is 6mb, which is fine but also not suitable for the biggest queries.

Are there any serverless services I can use to solve this problem? I do require custom domain / authentication. Some solution I thought of were:

  • Chunking request, which should work fine but I think 30s is still not very long. It is a temporary solution for now.

  • Using ALB as a "api" to trigger lambdas, which would fix the timeout, but response size limit is still 6mb.

  • Hosting my own API on a EC2/Container, which I can do but I like serverless solutions.

  • Using websockets, but it seems harder to attach existing apps to a WS compared to a REST api.

If somebody has some input would really appreciate it! Thanks in advance.

2 Answers

Hi there!

Both, the 30 sec timeout limit and 6mb payload size are hard limits in Lambda.

However, as a part of workaround, I would suggest a workaround around the same wherein you can process payloads that are over this 6MB limit.

== OPTION 1 ==

Instead of sending a huge payload directly to the Lambda function, you can upload the payload as an object to an S3 bucket, And then use the “Bucket Name” and “Object Name” to fetch the payload larger than 6 MB from s3 while executing the Lambda function using AWS SDKs to GET object from the bucket. This will allow the function to read the payload from the S3 object and process the data.

In case of sending out a response payload with size larger than 6 MB, I would recommend you to store the payload in S3 bucket and then pass the pre-signed URL generated for that payload stored inside S3 bucket to the end user/client in order to download the payload directly from S3 Bucket. For more details around the same please refer this documentation link below [1].

I would also like to share that on checking further around this, I found that there is an active feature request and is in the Lambda Service Team’s consideration (regarding the 6mb payload limit).

== OPTION 2 ==

As for alternatives to the Lambda Service, where you can have more control over the underlying environment, could be AWS Fargate.

AWS Fargate - a serverless version where containers are run on AWS-managed servers which you don't have to pay for - instead you pay for compute time [2].

Therefore as per your requirement, you can use ECS Fargate as an alternative. With AWS Fargate, you don't need to manage servers, handle capacity planning, or isolate container workloads for security. Fargate handles the infrastructure management aspects of your workload for you. You can schedule the placement of your containers across your cluster based on your resource needs, isolation policies, and availability requirements.

Hope this helps!




profile picture
answered 20 days ago
  • Thank you Ahmad! I considered option 1 but thought it wouldn't fit well, seems (like you said) more like a workaround then a proper solution. Fargate seems interesting, but I would still be bound to the 30s from api gateway. I think I even might be better off hosting an api on a docker container / ec2 right?

  • Hi again,

    You are right, API Gateway 30s timeout is indeed a hard limit. To overcome this:

    You could invoke your backend asynchronously [1]. This will mean that your backend will return a response to API Gateway straight away which will avoid timeout, and your backend can execute for longer than 29 seconds. In your backend you would return HTTP code 202 which means that the request has been accepted for processing, but the processing has not been completed. In the client you can implement polling logic to check the status of the backend.

    As for hosting the API on a container/EC2, this could certainly work, and would provide you with complete control over your environment configurations, if this is your use case.




To overcome Api GW limit, you can look at Application Load Balancer. You can increase timeout response from 60s to 4000s

As per lambda, you can evaluate Fargate to move your workload there. More details here:

Hope it helps!

profile picture
answered 20 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions