Log Retention Lambda

Question

Whenever we create a lambda using cdk, it also creates one log retention lambda. Is it by default ?
I understand that logs in cloudwatch will retain for long period of time. To save cost, we might want to set the retention time to lower date. Can we not set this as single/global config ? Creating a log retention lambda for every Lambda function that we create just causing some noise.

Can some expert in this forum help us - 
1. Whether we can delete this lambda  ? 
2. Are there a way to set the log related config in single place ?
3. Are there any CDK config to change so that we can avoid creating this lambda in future function ?

Please find the log retention lambda gist link
https://gist.github.com/gvnavin/71ec6310480834f632370b882ab6a581

Accepted Answer

When you specify logRetention in the Lambda function construct, CDK creates a custom resource, which is implemented using a Lambda function. You can delete the Lambda function, but that may cause issues if you need to delete the original stack. The Log retention function is shared across all functions in the same CDK app.

What you can do instead is create a Log group as part of the CDK application. When you create the Log group, you can define its retention period. The Log group must be named `/aws/lambda/`.

The following is an example:

```
from constructs import Construct
from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
    aws_logs as _logs
)

class CdkWorkshopStack(Stack):

def __init__(self, scope: Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

# Defines an AWS Lambda resource
        my_lambda = _lambda.Function(
            self, 'HelloHandler',
            function_name="HelloFunction",
            runtime=_lambda.Runtime.PYTHON_3_7,
            code=_lambda.Code.from_asset('lambda'),
            handler='hello.handler'
        )

log_group = _logs.LogGroup(
            self, "Logs",
            log_group_name="/aws/lambda/HelloFunction",
            retention=_logs.RetentionDays.ONE_DAY
        )
```

Answer

Hello, 
I can see 3 ways of doing this, all of them quite intrusive. 
All of them can read the "global config" from somewhere, for example, SSM parameter store, or have a default.

1. Create your own CDK construct that configures this automatically. 
2. Build your own Cloudformation Macro that configures this automatically.
3. A simple Lambda on a Cron job that finds any log group without a configured retention, fixes it and/or sends an alarm.

The perks with option 1 and 2 is that is scales better, and you can include more sugar in your platform with more standard config. But be aware that if you create the LogGroup with CloudFormation, it will (by default) be deleted with the Lambda. This is often what you want, but not always.

A problem with this is also that you need to enforce that Construct or Macro is being used, which will require some audit/automation (Maybe AWS Config is a good option). 
Another other problem with options 1 and 2 is that all your current Lambdas will need to do a CloudFormaiton Import to start managing your current logs.

Option 3 will probably leave you with empty zombie Log Groups that you have to deal with. It also has a "hacky magic" feeling to it. I guarantee that nobody will notice when it stops working in 10 years due to some changes in some API nobody knew you were using 😅.

Hope it helps!

Log Retention Lambda

Relevant content