AWS Cloudwatch - spamming of Fallback messages

0

Just recently we have a webserver cluster having disk space issues, and it's the awslogs.log file. These are all Ubuntu 20.04 LTS servers, and recently the drives are filling due to the awslogs.log file. These servers host a bunch of small websites, each vhost writes its own log, but now all the sites are spamming hundreds of warnings per minute with a warning such as;

2023-04-12 16:02:24,806 - cwlogs.push.reader - WARNING - 1266967 - Thread-4 - Fall back to current time: {'timestamp': 1681329744806, 'start_position': 1149160L, 'end_position': 1149310L}, reason: timestamp could not be parsed from message.

I commented all the sites in the config, so it is just the syslog (not doing it) and just one site and it starts right back. The odd thing is I am getting real data to cloudwatch, so it's a spam of warnings, then a success like this;

2023-04-12 16:02:18,931 - cwlogs.push.publisher - INFO - 1266967 - Thread-3 - Log group: websites, log stream: secure-website.access, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1681329733796, 'start_position': 1148714L, 'end_position': 1148865L}, 'fallback_events_count': 1, 'last_event': {'timestamp': 1681329733796, 'start_position': 1148714L, 'end_position': 1148865L}, 'source_id': 'e15231b897d06bbd6ca96b06bb81994e', 'num_of_events': 1, 'batch_size_in_bytes': 176}

Then more spamming. I am running the CloudWatch agent - 1.247358.0b252413, AWS CLI is aws-cli/2.11.11.

Apache Vhost logging looks like this; LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" combined LogFormat "%{X-Forwarded-For}i %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" proxy SetEnvIf X-Forwarded-For "^......." forwarded CustomLog "/var/log/apache2/secure.website.com-access.log" combined env=!forwarded CustomLog "/var/log/apache2/secure.website.com-access.log" proxy env=forwarded

And the AWS conf for this looks like; [/var/log/apache2/secure.website-access.log] datetime_format = %b %d %H:%M:%S file = /var/log/apache2/secure.website.com-access.log buffer_duration = 5000 log_stream_name = secure-website.access initial_position = start_of_file log_group_name = websites

Any help or suggestions is appreciated.

已提问 1 年前42 查看次数
没有答案

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则