Is there a way to download the full log file from AWS Cloudwatch Log groups?

0

Hello,

I'm trying to browse a giant log file on Cloudwatch and it's taking me forever to keep on clicking "load older logs". Is there any way I can download the whole log events as file?

Thanks

2 Answers
2

There are various open source command line tools, such as jorgebastida/awslogs and TylerBrock/saw, that you might find helpful.

AWS
John
answered 2 years ago
  • I was going to link to awslogs. Seems like a Johns thing (ha!).

0

FYI, we use awslogs. Requesting the entire log group in 1 go was SLOW.

So we created a process that splits up the request into specific periods (year, month, day, hour, minute, and second) and that generates a list of awslog calls, each one targeting a specific timeframe. The generated call's output is also piped through gzip and saved with the filename representing the log group and range ... and the last step ... all of that was run through parallel, effectively handling 10 requests at a time.

Running our wu command ("Workflow Utilities" - PHP / Symfony Console runner), wu logs:get-aws-logs nginx/access.log --split 3, we end up with a file of commands to run ...

mkdir -p '/Users/richardq/data/logs/aws/nginx-access' && awslogs get 'nginx/access.log' -G -S --no-color -s '2023-04-01 00:00:00' -e '2023-04-01 23:59:59' | gzip > '/Users/richardq/data/logs/aws/nginx-access/2023-04-01.log.gz'
mkdir -p '/Users/richardq/data/logs/aws/nginx-access' && awslogs get 'nginx/access.log' -G -S --no-color -s '2023-04-02 00:00:00' -e '2023-04-02 23:59:59' | gzip > '/Users/richardq/data/logs/aws/nginx-access/2023-04-02.log.gz'
mkdir -p '/Users/richardq/data/logs/aws/nginx-access' && awslogs get 'nginx/access.log' -G -S --no-color -s '2023-04-03 00:00:00' -e '2023-04-03 23:59:59' | gzip > '/Users/richardq/data/logs/aws/nginx-access/2023-04-03.log.gz'
mkdir -p '/Users/richardq/data/logs/aws/nginx-access' && awslogs get 'nginx/access.log' -G -S --no-color -s '2023-04-04 00:00:00' -e '2023-04-04 23:59:59' | gzip > '/Users/richardq/data/logs/aws/nginx-access/2023-04-04.log.gz'
...

That list runs of commands running via parallel, makes 1 file for each day that we've asked for on a log group. Multiple log groups can be asked for, effectively allowing us to do bulk retrieval. Overall, it seems faster for us.

One benefit really is that the local file is split and gzipped and so there are many Linux commands that can natively access GZipped content without needing to have the file unzipped.

Hopefully, this may help you.

profile picture
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions