AWS Lambda - Body Size is Too Large Error, but Body Size is Under Limit

0

Hello,

I am using a lambda function behind an AWS Gateway to service a REST API. In one endpoint I am getting "body size is too long" printed to the cloudwatch log.

The response I get is status code 502 with response body { "message": "Internal server error" }.

The response body size is 5622338 bytes (5.36 MB).

This is how I am calculating the response size (python 2.7):
out = {}
resp = urllib2.urlopen(req)
out.statusCode = resp.getcode()
out.body = resp.read()
print("num bytes: " + str(len(bytearray(out.body, 'utf-8'))))

The advertised response body max is 6MB. From what I understand, I should not be receiving the error.

Other information:
Duration: 22809.11 ms Billed Duration: 22810 ms Memory Size: 138 MB Max Memory Used: 129 MB Init Duration: 1322.71 m

https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

Any help would be appreciated.

Thank you in advance,
John

Edited by: gaffne67 on Apr 5, 2021 10:50 AM

Edited by: gaffne67 on Apr 5, 2021 10:55 AM

Edited by: gaffne67 on Apr 5, 2021 11:34 AM

Edited by: gaffne67 on Apr 22, 2021 12:13 PM
Noted that the function is behind an AWS Gateway.

asked 3 years ago7152 views
2 Answers
0

After further research I found that the lambda function errors out if the size of the response is 1,048,574 bytes (0.999998 MB) or more.

If the response is 1,048,573 bytes (0.999997 MB) or less it works.

This is how I am returning responses. I hard code the view function to return a bytearray of a specific size.
Ex.
return bytearray(1048573)

I turned on logging for the AWS Gateway Stage that I am using, and the following error is getting written to the log. It implies that the function is erroring out. Not the invocation of the function:
Lambda execution failed with status 200 due to customer function error: body size is too long.

It's my understanding that the AWS Lambda Functions have a max response size of 6MB and the AWS Gateways have a max response size of 10MB. Am I misunderstanding the limits?

AWS Lambda: Invocation payload response
https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

API Gateway: API Gateway quotas for configuring and running REST API -> Payload size
https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

answered 3 years ago
0

I created a new lambda function for python 3.7 and found that it returns a more descriptive error. I was able to determine that the lambda function adds around 1 MB to the size of the response after it leaves the lambda function which explains why it was erroring out in the lambda function, but not in the endpoint code. In one case it added .82MB and another it add .98MB. It seems to be based off the size of the response. I suspect the response is getting base64 encoded, URL encoded, or something similar. I did not find documentation that could answer this and the responses were not encoded in any way on the receiving end.

**Error Message returned by lambda function build with python 3.7: **
Response payload size (8198440 bytes) exceeded maximum allowed payload size (6291556 bytes).

Lambda Function Code:
import requests, base64, json

def lambda_handler(event, context):

headers = {...}  
url = ...  
  
response = requests.get(url, headers=headers)  
  
print (len(response.content))  
print (len(response.content.decode("utf-8")))  

  
return response.content.decode('UTF-8')  

Below I have the size of the responses when printed inside the lambda function and the size of the response as determined by lambda in the error message.

I used these two print statements to determine the size of the response and these two print statements would always return the same length. Response.content is a byte array so my thought process is that getting the length of it will return the number of bytes in the response:
print (len(response.content))
print (len(response.content.decode("utf-8")))

Ex 1.

Size when printed inside the lambda function:
7,165,488 (6.8335 MB)

Size as defined in the error message:
8,198,440 (7.8186 MB)

Extra:
1,032,952 (0.9851 MB)

Error Message:
Response payload size (8198440 bytes) exceeded maximum allowed payload size (6291556 bytes).

Ex 2.
Size when printed inside the lambda function:
5,622,338 (5.3619 MB)

Size as defined in the error message:
6,482,232 (6.1819 MB)

Extra:
859,894 (0.820059 MB)

Response payload size (6482232 bytes) exceeded maximum allowed payload size (6291556 bytes).

I have decided to lower the soft limit in the endpoint code by another 1 MB to prevent hitting the limit in the lambda function.

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions