AWS Cloudwatch - spamming of Fallback messages

0

Just recently we have a webserver cluster having disk space issues, and it's the awslogs.log file. These are all Ubuntu 20.04 LTS servers, and recently the drives are filling due to the awslogs.log file. These servers host a bunch of small websites, each vhost writes its own log, but now all the sites are spamming hundreds of warnings per minute with a warning such as;

2023-04-12 16:02:24,806 - cwlogs.push.reader - WARNING - 1266967 - Thread-4 - Fall back to current time: {'timestamp': 1681329744806, 'start_position': 1149160L, 'end_position': 1149310L}, reason: timestamp could not be parsed from message.

I commented all the sites in the config, so it is just the syslog (not doing it) and just one site and it starts right back. The odd thing is I am getting real data to cloudwatch, so it's a spam of warnings, then a success like this;

2023-04-12 16:02:18,931 - cwlogs.push.publisher - INFO - 1266967 - Thread-3 - Log group: websites, log stream: secure-website.access, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1681329733796, 'start_position': 1148714L, 'end_position': 1148865L}, 'fallback_events_count': 1, 'last_event': {'timestamp': 1681329733796, 'start_position': 1148714L, 'end_position': 1148865L}, 'source_id': 'e15231b897d06bbd6ca96b06bb81994e', 'num_of_events': 1, 'batch_size_in_bytes': 176}

Then more spamming. I am running the CloudWatch agent - 1.247358.0b252413, AWS CLI is aws-cli/2.11.11.

Apache Vhost logging looks like this; LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" combined LogFormat "%{X-Forwarded-For}i %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" proxy SetEnvIf X-Forwarded-For "^......." forwarded CustomLog "/var/log/apache2/secure.website.com-access.log" combined env=!forwarded CustomLog "/var/log/apache2/secure.website.com-access.log" proxy env=forwarded

And the AWS conf for this looks like; [/var/log/apache2/secure.website-access.log] datetime_format = %b %d %H:%M:%S file = /var/log/apache2/secure.website.com-access.log buffer_duration = 5000 log_stream_name = secure-website.access initial_position = start_of_file log_group_name = websites

Any help or suggestions is appreciated.

asked a year ago42 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions