- Newest
- Most votes
- Most comments
I can't talk to why the limit hasn't been increased. But the most common solution is to place large objects into S3 and then return a presigned URL to the caller to access that object. You could also take advantage of CloudFront and the global AWS backbone network just by using those services.
You need to make sure that the credentials used to generate the presigned URL are long-lived enough for the client to retrieve the object - if it is a one-time retrieval then the credentials from the Lambda function are usually fine; but many implementations use a separate set of credentials so that if the Lambda function credentials expire before the client has retrieved the object then it still works.
You'd also presumably need to clean up the objects in S3 afterwards - this can easily accomplished with bucket lifecycle policies.
You could implement a chunked transfer mechanisms using API Gateway and Lambda; it would require you to store the object somewhere permanent anyway (because there's no guarantee that separate calls to retrieve chunks would hit the same Lambda invocation) - so there's a bunch of work to be done if you went down that path. I wouldn't recommend it but it can be done.
Relevant content
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
This solution sounds like the right approach.
A note on the credentials used to generate the S3 pre-signed URLs: As @Brettski mentions, if the credentials expire before the client has retrieved the object, this mechanism will result in your client failing to retrieve the result. Since you as a customer can't control the lifecycle of Lambda functions, you don't have control over how long these credentials will last. It could expire a second after the S3 presigned URL is generated, if the Lambda container is recycled.
Because of this, a separate set of credentials must be used in this case.
The chunked transfer mechanism I'm looking for is an equivalent to
Transfer-Encoding: chunked, a basic HTTP feature with no need for separate calls.It can be awkward to determine whether a response will exceed the 6MB limit and then require a full roundtrip redirect to the client (because API Gateway does not to my knowledge support an equivalent to Nginx's X-Accel-Redirect.)
I guess I just want to be able to run an HTTP server in a scale to zero container as I can on Cloud Run. It's frustrating there's not yet an equivalent in AWS.