Add additional fields to Logs Insights from python logger

Question

Hi.

I'm filtering my logs from a Fargate container using Cloud Watch Insights with something like this

```
fields @timestamp, @message
| sort @timestamp desc
| filter @message like /(ERROR:root:)/
| limit 20

```

I know that lambda add additionals fields like @requestId, but not sure if is possible create custom additional fields in some way with the python logger, for example the log level and do something like this:

```
fields @timestamp, @level, @message
| sort @timestamp desc
| filter @level = 'ERROR'
| limit 20
```

It's possible to add custom fields like lambda does?

alex_lykasov · Answer

Cloudwatch Logs Insights has fields auto discovery feature. It automatically discovers fields from some of AWS services, like Lambda as you mentioned. It also discovers all fields in JSON documents. So one of the solutions could be to output logging messages in JSON format. If you are using python, you can configure your logger to use `python-json-logger`, or implement json logging yourself.

Let's say we have this kind of messages in Cloudwatch logs:
```
{
    "levelname": "ERROR",
    "message": "Some kind of exception occurred",
    "funcName": "function_name",
    "filename": "app.py"
}
```

Then you can filter by json fields in CW Logs Insights like this:
```
fields @timestamp, @message
| filter levelname="ERROR" and funcName="function_name"
| sort @timestamp desc
| limit 20
```

You can also have nested json in your log messages and filter based on those. Another important part about json logs is to have consistent schema for logging messages across the whole application.

* More about fields discovery and JSON logs - https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html
* Python JSON logger - https://pypi.org/project/python-json-logger/

Add additional fields to Logs Insights from python logger

Relevant content