Add additional fields to Logs Insights from python logger



I'm filtering my logs from a Fargate container using Cloud Watch Insights with something like this

fields @timestamp, @message
| sort @timestamp desc
| filter @message like /(ERROR:root:)/
| limit 20

I know that lambda add additionals fields like @requestId, but not sure if is possible create custom additional fields in some way with the python logger, for example the log level and do something like this:

fields @timestamp, @level, @message
| sort @timestamp desc
| filter @level = 'ERROR'
| limit 20

It's possible to add custom fields like lambda does?

asked a year ago510 views
1 Answer

Cloudwatch Logs Insights has fields auto discovery feature. It automatically discovers fields from some of AWS services, like Lambda as you mentioned. It also discovers all fields in JSON documents. So one of the solutions could be to output logging messages in JSON format. If you are using python, you can configure your logger to use python-json-logger, or implement json logging yourself.

Let's say we have this kind of messages in Cloudwatch logs:

    "levelname": "ERROR",
    "message": "Some kind of exception occurred",
    "funcName": "function_name",
    "filename": ""

Then you can filter by json fields in CW Logs Insights like this:

fields @timestamp, @message
| filter levelname="ERROR" and funcName="function_name"
| sort @timestamp desc
| limit 20

You can also have nested json in your log messages and filter based on those. Another important part about json logs is to have consistent schema for logging messages across the whole application.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions