CloudWatch Logs sending to Firehose is missing data

Hi,

Did you take the following note into account ?

It can lead to a different number of log events and files written to S3. You need to separate the combined events on S3 side if you want to match the numbers.

Important
Currently, Firehose does not support the delivery of CloudWatch Logs to Amazon
 OpenSearch Service destination because Amazon CloudWatch combines multiple 
log events into one Firehose record and Amazon OpenSearch Service cannot accept 
multiple log events in one record. As an alternative, you can consider Using subscription 
filter for Amazon OpenSearch Service in CloudWatch Logs.

See https://docs.aws.amazon.com/firehose/latest/dev/writing-with-cloudwatch-logs.html

Best,

Didier

There are several steps you can take to troubleshoot this issue and ensure all data is being sent from CloudWatch Logs to Firehose and then to S3.

First, check the CloudWatch metrics for your Firehose delivery stream. Look at the IncomingBytes and IncomingRecords metrics under the Monitoring tab in the Firehose console. This will help you confirm if the Firehose stream is receiving data from CloudWatch Logs.

Next, verify the IAM role permissions. Ensure that the IAM role specified in your Firehose stream has the necessary permissions to access your S3 bucket and CloudWatch log groups. The role should also have permissions to create and write to CloudWatch log streams for error logging.

Enable CloudWatch Logs error logging on the Firehose delivery stream to get more detailed information about any potential errors occurring during the data transfer process.

Check the configuration of your Firehose delivery stream to ensure it's correctly set up to deliver data to your S3 bucket. Verify that the destination settings are accurate.

It's important to note that there is a known limitation with CloudWatch Logs and Firehose integration. CloudWatch Logs combines multiple log events into one Firehose record, which can cause issues with certain destinations. However, this is typically a problem when sending to destinations like Amazon OpenSearch Service, not S3.

If you're still experiencing discrepancies after checking these points, you may want to review the CloudWatch Logs retention period and ensure you're not losing any data due to log expiration. Also, consider the possibility of data transformation or filtering occurring somewhere in the pipeline that could be affecting the number of events.

If the issue persists, you might want to consider using CloudWatch Logs Insights to query your log data and compare it with what's being delivered to S3 to identify any patterns or specific log streams that might be affected.
Sources
Troubleshooting Amazon S3 - Amazon Data Firehose
Troubleshooting Amazon OpenSearch Service - Amazon Data Firehose
Troubleshoot a CloudWatch stream that doesn't push metrics | AWS re:Post