How to group multiple log stream under same log group

CloudWatch Logs > Log Group(s) > Log Stream(s) > [Log Event]

We cannot have directories within a log stream i.e. the following level of categorization is not possible:

AWSService_log_group
 /service_#1
     /file_path/logs
        /xxxx.log

However, you can have multiple log streams in a log group i.e. the following is possible assuming "xxxx.log" is not a sub-directory but directly the log events/entries:

Instance_log_group
  /instance_#1_log_a
      /xxxx.log (log entries)
  /instance_#1_log_b
      /xxxx.log
  /instance_#2_log_a
      /xxxx.log
  /instance_#2_log_b
      /xxxx.log

To publish log events to separate log streams, you can specify this in the "log_collected" section such as following where 2 log streams in the "test.log" log group will be created. Using the log_stream_name field-

"logs_collected": {
           "files": {
               "collect_list": [
                   {
                       "file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\amazon-cloudwatch-agent.log",
                       "log_group_name": "test.log",
                       "log_stream_name": "my_log_stream_name_1_{instance_id}"
                   },
                   {
                       "file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\test.log",
                       "log_group_name": "test.log",
                       "log_stream_name": "my_log_stream_name_2_{instance_id}"
                   }
               ]
           }

Adding a chunk from documentation [1] here for better understanding:

log_stream_name – Optional. Specifies what to use as the log stream name in CloudWatch Logs. As part of the name, you can use {instance_id}, {hostname}, {local_hostname}, and {ip_address} as variables within the name. {hostname} retrieves the hostname from the EC2 metadata, and {local_hostname} uses the hostname from the network configuration file.

If you omit this field, the value of the log_stream_name parameter in the global logs section is used. If that is also omitted, the default value of {instance_id} is used. If a log stream doesn't already exist, it's created automatically.

[1] Manually create or edit the CloudWatch agent configuration file - CloudWatch agent configuration file: Logs section - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html#CloudWatch-Agent-Configuration-File-Logssection

---

Using the above method, each instance can have its own log stream for the type of logs that it is pushing to CloudWatch. But, there cannot be sub-directories beyond a log stream in CloudWatch. You can make use of the variables such as {instance_id} in the configuration, for log-group names or log-stream names to categorize. The documentation link provided above explains this in detail.