application log management with AWS cloud watch agent

Question

I am puzzled by how to effectively transfer application logs to CloudWatch. I have 12 microservices, each of which generates five different types of logs located at /var/log/application/service-name. My application is deployed on an AWS ECS cluster, consisting of two large instances capable of hosting all 12 microservices, with each instance accommodating six services.

The primary challenge lies in managing the logs. I aim to organize them such that each type of log resides within a separate log group, and each of the 12 services has its logs stored as individual log streams. In essence, this necessitates creating log groups and 12 log streams per log group.

I have attempted to address this issue by installing the CloudWatch agent and creating a log configuration file. However, I encountered difficulties in properly segregating the logs.

Ideally, I require log group names structured as env_name-log_type.log and log stream names formatted as service_name-instance_id.log.

Please provide suggestions on how to achieve this segmentation.

Answer

To achieve the desired log segmentation structure, you can configure the CloudWatch agent to send logs to CloudWatch Logs using log groups and log streams.

Some suggestions:

Configure the CloudWatch agent on each EC2 instance using a configuration file that specifies the log file locations and log group/stream formats.

For the log group name, use the format 
env_name-log_type.log
 (e.g. 
prod-access.log
)

For each microservice, define a log stream with the name format 
service_name-instance_id.log
 (e.g. 
paymentservice-i-0123456.log
)

The agent will collect logs from the specified files and write them to CloudWatch Logs using the defined log groups and streams.

You can define multiple outputs in the configuration to send the same logs to other destinations like S3 if required.

Logs from all instances will be organized by service and type in the CloudWatch console for easy analysis and troubleshooting.

Let me know if any part needs more explanation or if you have additional questions!

application log management with AWS cloud watch agent

Relevant content