Add custom fields to structured logs in CloudWatch Agent via configuration

Question

Our application utilizes structured logging to disk (JSON files). We are migrating from an ELK solution to CloudWatch. We are currently utilizing filebeat to push logs to ELK. In the configuration for filebeat you could specify fields per instance via configuration like this:

```
fields.client: 999
fields.env: PROD
fields.loc: US
```

Is there a similar way to append fields to each message like filebeat does in CloudWatch Agent? Is there perhaps another solution we can use that will fulfill the same feature? We do not want to have to modify all our applications to add these fields to each message.

Sagar_T · Answer

Hi there,

Sure! You can add custom fields to structured logs in CloudWatch Agent via the configuration. Here's how:

1) Open the CloudWatch Agent configuration file in a text editor. The configuration file is typically located at '/etc/aws/cloudwatch-agent/config.json'.

2) Look for the logs section in the configuration file, which should look something like this:
________________________
{
  "logs": {
    "input_logs": [
      {
        "input_name": "MyAppLogs",
        "input_type": "log",
        "input_path": "/var/log/myapp/app.log",
        "input_format": "json"
      }
    ]
  }
}
________________________

3) Add a new input_log object to the input_logs array, with the name and path of the custom log file you want to collect. For example:
________________________
{
  "logs": {
    "input_logs": [
      {
        "input_name": "MyAppLogs",
        "input_type": "log",
        "input_path": "/var/log/myapp/app.log",
        "input_format": "json"
      },
      {
        "input_name": "CustomLog",
        "input_type": "log",
        "input_path": "/var/log/custom/log.json",
        "input_format": "json"
      }
    ]
  }
}
________________________

4) Add the custom fields you want to collect in the input_log object, using the input_fields property. For example:
________________________
{
  "logs": {
    "input_logs": [
      {
        "input_name": "MyAppLogs",
        "input_type": "log",
        "input_path": "/var/log/myapp/app.log",
        "input_format": "json"
      },
      {
        "input_name": "CustomLog",
        "input_type": "log",
        "input_path": "/var/log/custom/log.json",
        "input_format": "json",
        "input_fields": [
          {
            "name": "custom_field_1",
            "type": "string"
          },
          {
            "name": "custom_field_2",
            "type": "integer"
          }
        ]
      }
    ]
  }
}
________________________

Save the configuration file and restart the CloudWatch Agent service for the changes to take effect.

Once you've made these changes, the CloudWatch Agent will collect the custom fields you specified in the input_fields property, along with the log data from the custom log file. You can then view the custom fields in CloudWatch Logs Insights, along with the rest of the log data.

I hope this helps!

Add custom fields to structured logs in CloudWatch Agent via configuration

Add your answer

Relevant content