Are we able to export only parts of the Amazon CloudWatch logs to Amazon S3?

Question

A customer wants to export only parts of the logs from CloudWatch to S3, is there a way to do that?

Reason is that the logGroup is very big and they are  only interested in the data for the last months.

Is there a way from AWS to delete older log streams periodically?
To save some costs on storage?

Accepted Answer

The pattern you describe is quite common, what you want to do is create a subscription filter for your cloudwatch log group.

Ideally this is the pattern [CW Log Group Filter] -->[Kinesis Firehose] -->[Delivery Stream] -->[S3]

Read the following guide for an example:

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#FirehoseExample

As to your question on a way to periodically delete log streams, CW Log Group retention periods are configurable between 1 day up to 10 years.

Relevant content