- Newest
- Most votes
- Most comments
The only way in which you can have code running after you return the response from the handler is using Lambda extensions. You will need to create an extension that will wait for the main function to send it the file (it could be done using an internal socket or using a file in /tmp). Once the function sends the file to the extension it can return. The extension can then save the file to S3.
You can find some deep dive session here.
This isn't going to seem helpful, but: Things take time and sometimes there aren't good solutions to that. However, some suggestions:
CPU performance in Lambda functions increases as memory allocation does. So if you allocate more memory to your Lambda function (if even you don't use it) it will get more CPU allocated to it and (probably) execute faster. There will be a balancing point for that where adding more memory doesn't decrease execution time.
You may find that other storage types may be faster than S3 but you will need to test. You could try using EFS with your Lambda function but that will add some cost to your solution but it may meet your performance requirements. You might choose to store the request object in EFS, send a SQS message which triggers another Lambda function to copy that to S3 for longer-term (and less expensive) storage; then delete the file from EFS.
If there is a challenge with cold start timing (it doesn't appear to be that way but just in case) then consider using Lambda Provisioned Concurrency noting again that there is a cost there.
Thanks Brettski for your suggestions.
Yes we have increased memory to 2048M although only around 128M used, and it didn't really improve the S3 operation and SNS event publishing time that much. We have implemented provisioned concurrency, it does eliminate the lambda initialisation time when it cold starts. We haven't tried EFS but tried ElastiCache. One problem with ElastiCache is, a provisioned lambda is not maintaining TCP connections so upon first API call lambda needs to reconnect to ElastiCache which makes it almost as slow as S3. Anyway the ElastiCache is ruled out by the mgmt as it is not server-less infra.
The delay you're experiencing connecting to ElastiCache is also going to be present connecting to S3 and SNS; those services also rely on TCP connections being created and TLS being negotiated; both are going to lead to delays; as above: things take time and those things don't really change that much with increased CPU.
Hi Brettski, you are right that connecting to ElastiCache / S3 / SNS are over the same TCP and TLS, the difference is that when lambda instance is warm, the TCP connection to ElastiCache is maintained. I've uploaded two XRAY trace for your info.
It is possible to upload files to S3 directly from API Gateway without having a lambda in between. The following https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-upload-image-s3/ shows how that can be done.
If you have to use a Lambda for other reasons and want to send the data to S3 asynchronously, you can use one of the following approaches if your file size max is 1 MB.
a) From the Lambda, write to Kinesis Data Firehose and that in turn writes to S3 b) From the Lambda, write to Kinesis Data Streams and have another Lambda pick up from the Kinesis Data Streams and write to S3 c) From the Lambda, write to Kinesis Data Streams and have Kinesis Data Firehose read from Kinesis Data Streams and write to S3
Thanks Indranil. The scenario in my question is a simplified version, our business requirements are more than just uploading the payload. I will try Firehose when I get a chance but I doubt it would be faster than writing the file into S3 directly.
Use the Lambda to pass the request to a Step Function and then return the response immediately.
Relevant content
- asked 7 months ago
- Accepted Answerasked a year ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 3 years ago
Thanks Uri. Will give it a go whenever I get a chance.