Cloud Watch agent stopped sending logs after clearing log file

0

I have a working Cloudwatch agent on my EC2 (belonging to an ASG+ALB), streaming my log file (the java tomcat log file catalina.out) to my Cloudwatch log group. I would like to regularly clear my log files on the EC2 to avoid getting a full disk.

When I manually make a log file clear (echo "" > catalina.out), the agent continues to send incoming logs from this file to the log group. But when I'm asking crontab to do it for me, the Cloudwatch agent stops sending the logs. Why? The same user ran the two operations and my file permissions didn't change. This test was done with the old Cloudwatch agent running with Python. I've tried with the latest Cloudwatch agent (amazon-cloudwatch-agent) and, in this case, the clear (echo "" > catalina.out) is enough to break the log streaming. I've tried to stop the Cloudwatch agent before clearing and restarting after, but the issue remains.

How to find out what's going on and where the issue could be?

asked a month ago442 views
1 Answer
2
Accepted Answer

Hi THere

Instead of clearing the log file with the echo command, id recommend implementing something like logrotate or one of the supported methods below:

From https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html#agent-faq

The following file rotation mechanisms are supported:

  1. Renaming existing log files with a numerical suffix, then re-creating the original empty log file. For example, /var/log/syslog.log is renamed /var/log/syslog.log.1. If /var/log/syslog.log.1 already exists from a previous rotation, it is renamed /var/log/syslog.log.2. (logrotate will do this for you)
  2. Truncating the original log file in place after creating a copy. For example, /var/log/syslog.log is copied to /var/log/syslog.log.1 and /var/log/syslog.log is truncated. There might be data loss for this case, so be careful about using this file rotation mechanism.
  3. Creating a new file with a common pattern as the old one. For example, /var/log/syslog.log.2014-01-01 remains and /var/log/syslog.log.2014-01-02 is created.

You might also find the exact reason that the logs stop streaming by examining the CloudWatch agent log file. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/troubleshooting-CloudWatch-Agent.html#CloudWatch-Agent-troubleshooting-loginfo But I would still try and stick with one of the supported rotation mechanisms to ensure you do not lose data.

profile pictureAWS
EXPERT
Matt-B
answered a month ago
profile picture
EXPERT
reviewed a month ago
  • Thanks for your help. The solution was to use logrotate instead of directly writing a cron. Delete or move the catalina.out file wasn't possible as would break the tomcat logs, even if we created a new empty file after. That's why I used cron to follow the 2nd point you've raised here. Don't know why logrotate made it work and not cron, but at least it's working!

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions