- Newest
- Most votes
- Most comments
The PutLogsEvents API has a limit of 800/1500 requests per second, depending on the region. The limit can be increased. The API supports batch operations, so you can reduce the number of requests to the number of invocations (no need to send each line by itself, you can batch them and make a single API call).
You could build this mechanism inside your functions (i.e., add each event to an array and at the end of the invocation, call the API), or, you can create your own Lambda extension that will receive all log events, buffer them and sends them in one API call. Note, that if you use the extension, the externalized logs will show in two log groups (the function's group and the exposed group). If you build the mechanism yourself, you can decide to send the exposed logs only to the external log group.
Another option is to send all events to the default log group and then create a subscriber Lambda that will move only the expose logs to the other log group. This way, it is external to your function so you do not need to implement it in each one. You create a single function and you subscribe it to all the relevant log groups.
Relevant content
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated a year ago
Thanks for the answer Uri :) In terms of overall cost do you have a picture which solution would be more cost efficient? Lambda extension on each function that writes to CW vs a log stream via a subscriber Lambda. I assume Lambda extension is more cost-efficient.
Eventually you want the events to get to the per customer log group, which means that you will need to pay for that. The question is what other costs you have withe the different solutions.
I think the cheapest would be to buffer in your function and send a single event using PutLogEvents, This way you don't need to pay for the cost of these log messages in the function's log group. The second one would be to use an extension. In this option you pay for ingesting the messages to both log groups, and no other cost. I think the most expensive would be to use the Lambda subscriber, because you also need to run an additional Lambda function to copy the messages from the function's log group to the customer's log group.
Note that by default log messages are saved for ever, so don't forget to set a retention on the different log groups.
One caveat I'm seeing with this approach is that sometimes logs are not delivered to the extension for various reasons, which causes delays from 1min to 10mins or even dropped logs. So probably the streaming solution to a different Lambda from native logs is more robust, which comes at a bit higher cost. Let me know if this is not the correct understanding Uri :)
Are you catching the shutdown event in the extension? You should catch it and use it to flush all accumulated logs.