How to use regex in CloudWatch agent to exclude specific files from a log group?

Question

Hello AWS community,

I am currently setting up CloudWatch logs on my EC2 instance using the CloudWatch agent. I have the following use case, but I'm facing difficulties filtering files using regex:

**# My goal:**

I have two categories of log files:

Files that match the pattern *_error_yyyymmdd.log, which I want to send to a log group called `/logs/error`

and Files that match the pattern *_web_error_yyyymmdd.log are sent to a different log group `/logs/web_error`

I do not want the logs from web_error be included in error log group.

I try to use regular expression but not work. Does cloud watch agent configuration file support regex for file path? If not, is there any simple way to implement this?

Thank you!

Riku_Kobayashi · Accepted Answer

Hello.

As far as I know, only the asterisk wildcard can be used in the Logs section of CloudWatch Agent.  
Therefore, I think the problem can be solved by creating separate directories for "*_error_yyyymmdd.log" and "*_web_error_yyyymmdd.log".  
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html#CloudWatch-Agent-Configuration-File-Logssection

In other words, as shown below, it is possible to set the file path in the CloudWatch Agent settings, so I think you can deal with this by changing the output file path on the application side.  
```
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/httpd/error/logs/*_error_*.log",
            "log_group_name": "/logs/error",
            "log_stream_name": "{instance_id}"
          },
          {
            "file_path": "/var/log/httpd/web_error/logs/*_web_error_*.log",
            "log_group_name": "/logs/web_error",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
```

How to use regex in CloudWatch agent to exclude specific files from a log group?

Add your answer

Relevant content