AWS Cloudwatch - spamming of Fallback messages

0

Just recently we have a webserver cluster having disk space issues, and it's the awslogs.log file. These are all Ubuntu 20.04 LTS servers, and recently the drives are filling due to the awslogs.log file. These servers host a bunch of small websites, each vhost writes its own log, but now all the sites are spamming hundreds of warnings per minute with a warning such as;

2023-04-12 16:02:24,806 - cwlogs.push.reader - WARNING - 1266967 - Thread-4 - Fall back to current time: {'timestamp': 1681329744806, 'start_position': 1149160L, 'end_position': 1149310L}, reason: timestamp could not be parsed from message.

I commented all the sites in the config, so it is just the syslog (not doing it) and just one site and it starts right back. The odd thing is I am getting real data to cloudwatch, so it's a spam of warnings, then a success like this;

2023-04-12 16:02:18,931 - cwlogs.push.publisher - INFO - 1266967 - Thread-3 - Log group: websites, log stream: secure-website.access, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1681329733796, 'start_position': 1148714L, 'end_position': 1148865L}, 'fallback_events_count': 1, 'last_event': {'timestamp': 1681329733796, 'start_position': 1148714L, 'end_position': 1148865L}, 'source_id': 'e15231b897d06bbd6ca96b06bb81994e', 'num_of_events': 1, 'batch_size_in_bytes': 176}

Then more spamming. I am running the CloudWatch agent - 1.247358.0b252413, AWS CLI is aws-cli/2.11.11.

Apache Vhost logging looks like this; LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" combined LogFormat "%{X-Forwarded-For}i %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" proxy SetEnvIf X-Forwarded-For "^......." forwarded CustomLog "/var/log/apache2/secure.website.com-access.log" combined env=!forwarded CustomLog "/var/log/apache2/secure.website.com-access.log" proxy env=forwarded

And the AWS conf for this looks like; [/var/log/apache2/secure.website-access.log] datetime_format = %b %d %H:%M:%S file = /var/log/apache2/secure.website.com-access.log buffer_duration = 5000 log_stream_name = secure-website.access initial_position = start_of_file log_group_name = websites

Any help or suggestions is appreciated.

posta un anno fa42 visualizzazioni
Nessuna risposta

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande