How to group multiple log stream under same log group

0

I have several EC2 instances (Windows servers) which include several applications with quite an amount of log files in each. Is it possible to push logs so that they would be arranged in the form of a logical directory structure in the CloudWatch? (like shown below):

Instance_log_group
  /instance_#1_log_a
      /xxxx.log
  /instance_#1_log_b
      /xxxx.log
  /instance_#2_log_a
      /xxxx.log
  /instance_#2_log_b
      /xxxx.log
AWSService_log_group
 /service_#1
     /file_path/logs
        /xxxx.log
  /service_#2
     /file_path/logs
        /xxxx.log
        /xxxx.log

Basically, how to configure or modify my Cloudwatch configuration file (configured using SSM) - to have more than one log stream under the same log group.

2 Answers
0
Accepted Answer

CloudWatch Logs > Log Group(s) > Log Stream(s) > [Log Event]

We cannot have directories within a log stream i.e. the following level of categorization is not possible:

AWSService_log_group
 /service_#1
     /file_path/logs
        /xxxx.log

However, you can have multiple log streams in a log group i.e. the following is possible assuming "xxxx.log" is not a sub-directory but directly the log events/entries:

Instance_log_group
  /instance_#1_log_a
      /xxxx.log (log entries)
  /instance_#1_log_b
      /xxxx.log
  /instance_#2_log_a
      /xxxx.log
  /instance_#2_log_b
      /xxxx.log

To publish log events to separate log streams, you can specify this in the "log_collected" section such as following where 2 log streams in the "test.log" log group will be created. Using the log_stream_name field-

"logs_collected": {
           "files": {
               "collect_list": [
                   {
                       "file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\amazon-cloudwatch-agent.log",
                       "log_group_name": "test.log",
                       "log_stream_name": "my_log_stream_name_1_{instance_id}"
                   },
                   {
                       "file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\test.log",
                       "log_group_name": "test.log",
                       "log_stream_name": "my_log_stream_name_2_{instance_id}"
                   }
               ]
           }

Adding a chunk from documentation [1] here for better understanding:

log_stream_name – Optional. Specifies what to use as the log stream name in CloudWatch Logs. As part of the name, you can use {instance_id}, {hostname}, {local_hostname}, and {ip_address} as variables within the name. {hostname} retrieves the hostname from the EC2 metadata, and {local_hostname} uses the hostname from the network configuration file.

If you omit this field, the value of the log_stream_name parameter in the global logs section is used. If that is also omitted, the default value of {instance_id} is used. If a log stream doesn't already exist, it's created automatically.

[1] Manually create or edit the CloudWatch agent configuration file - CloudWatch agent configuration file: Logs section - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html#CloudWatch-Agent-Configuration-File-Logssection


Using the above method, each instance can have its own log stream for the type of logs that it is pushing to CloudWatch. But, there cannot be sub-directories beyond a log stream in CloudWatch. You can make use of the variables such as {instance_id} in the configuration, for log-group names or log-stream names to categorize. The documentation link provided above explains this in detail.

profile pictureAWS
SUPPORT ENGINEER
answered a year ago
0

Thank you @shreyas, this will help and I will try this once.

Related to this, I have configured the CloudWatch agent configuration file in SSM and pushed five application-related logs from the EC2 instance as below,

  1. Out of five only the first three are reflected in CloudWatch
  2. Windows metrics are also not reflected in CloudWatch
	"logs": {
		"logs_collected": {
			"files": {
				"collect_list": [
					{
						"file_path": "C:\\Program Files\\*****\\*****\\*****\\*****\\name*.log",
						"log_group_name": "group1",
						"log_stream_name": "{instance_id}",
						"retention_in_days": 60
					},
					{
						"file_path": "C:\\Program Files\\*****\\*****\\*****\\*****\\name1.log",
						"log_group_name": "group2",
						"log_stream_name": "{instance_id}",
						"retention_in_days": 60
					},
					{
						"file_path": "C:\\Program Files\\*****\\*****\\*****\\*****\\name2*.log",
						"log_group_name": "group3",
						"log_stream_name": "{instance_id}",
						"retention_in_days": 60
					},
					{
						"file_path": "C:\\Program Files\\*****\\*****\\*****\\*****\\name3*.log",
						"log_group_name": "group4",
						"log_stream_name": "{instance_id}",
						"retention_in_days": 60
					},
					{
						"file_path": "C:\\Program Files\\*****\\*****\\*****\\**.log",
						"log_group_name": "group5",
						"log_stream_name": "{instance_id}",
						"retention_in_days": 60
					}
				]
			},
			"windows_events": {
				"collect_list": [
					{
						"event_format": "xml",
						"event_levels": [
							"VERBOSE",
							"INFORMATION",
							"WARNING",
							"ERROR",
							"CRITICAL"
						],
						"event_name": "Windows-system",
						"log_group_name": "group6",
						"log_stream_name": "{instance_id}",
						"retention_in_days": 30
					}
				]
			}
		}
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions