Just recently we have a webserver cluster having disk space issues, and it's the awslogs.log file. These are all Ubuntu 20.04 LTS servers, and recently the drives are filling due to the awslogs.log file. These servers host a bunch of small websites, each vhost writes its own log, but now all the sites are spamming hundreds of warnings per minute with a warning such as;
2023-04-12 16:02:24,806 - cwlogs.push.reader - WARNING - 1266967 - Thread-4 - Fall back to current time: {'timestamp': 1681329744806, 'start_position': 1149160L, 'end_position': 1149310L}, reason: timestamp could not be parsed from message.
I commented all the sites in the config, so it is just the syslog (not doing it) and just one site and it starts right back. The odd thing is I am getting real data to cloudwatch, so it's a spam of warnings, then a success like this;
2023-04-12 16:02:18,931 - cwlogs.push.publisher - INFO - 1266967 - Thread-3 - Log group: websites, log stream: secure-website.access, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1681329733796, 'start_position': 1148714L, 'end_position': 1148865L}, 'fallback_events_count': 1, 'last_event': {'timestamp': 1681329733796, 'start_position': 1148714L, 'end_position': 1148865L}, 'source_id': 'e15231b897d06bbd6ca96b06bb81994e', 'num_of_events': 1, 'batch_size_in_bytes': 176}
Then more spamming. I am running the CloudWatch agent - 1.247358.0b252413, AWS CLI is aws-cli/2.11.11.
Apache Vhost logging looks like this;
LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" proxy
SetEnvIf X-Forwarded-For "^......." forwarded
CustomLog "/var/log/apache2/secure.website.com-access.log" combined env=!forwarded
CustomLog "/var/log/apache2/secure.website.com-access.log" proxy env=forwarded
And the AWS conf for this looks like;
[/var/log/apache2/secure.website-access.log]
datetime_format = %b %d %H:%M:%S
file = /var/log/apache2/secure.website.com-access.log
buffer_duration = 5000
log_stream_name = secure-website.access
initial_position = start_of_file
log_group_name = websites
Any help or suggestions is appreciated.
