What's the best way to organise and archive a large number of Cloudwatch Log Groups?

Question

We currently have around 150 Cloudwatch Log Groups in each of our 8 application accounts (within an AWS Organisation). We want to archive them to S3.

And do this in an automated fashion as we tried getting a junior to do it and they just started crying (joking ... no junior involved ... but the crying was real once I realised that doing this by hand would become my full time job!!!).

I've read https://medium.com/dnx-labs/exporting-cloudwatch-logs-automatically-to-s3-with-a-lambda-function-80e1f7ea0187, which is a little out of date now, but the big issue here is that there seems to be restrictions on the number of exports that can be run simultaneously.

Whilst we use Terraform for our IaC, not all of our log groups are created or managed by Terraform. Some of the applications seem to create their own, and so tagging is currently inconsistent. It can be addressed, but we're happy for all the log groups to be archived.

So what is the current best way to do bulk archiving?

For extra points, we have been contemplating archiving the logs into 1 bucket, within our DevOps account, rather than a bucket per account. The logs are NOT for our client's usage, just our own.

We "inherited" the multiple accounts structure and are looking to see what's required to amalgamate them but that's not on the radar for this year.

I can happily write shell scripts if Lambdas are needed.

Any comments, suggestions, recommended reading, etc. ALL gratefully received.

Thank you for reading.

Answer

The solution would depend on how quickly you needed the CloudWatch Logs to be exported to S3.

If you can run on a schedule, say every 4 or 5 hours, I'd suggest the solution listed in the article you suggested, but rather than triggering the log exports directly from the initial lambda, use the initial lambda to publish a message to SNS with details of the log group to be exported and then poll the SQS to retrieve the details and export that group. Configure the poller to retrieve the messages individually with a polling period set appropriately to ensure the exports are completed before the next message in the queue is completed.

If you want more immediate exports (bearing in mind this will create more objects in s3), look at creating a subscription filter on the cloudwatch log group (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html), then feeding into the solution above i.e. calling a lambda to publish the results to sns and then process in the same way.

What's the best way to organise and archive a large number of Cloudwatch Log Groups?

Relevant content