Is it possible to get a single & sorted exported log data from CloudWatch Logs export task?
Currently I use the CreateExportTask API to backup my log data.
The problem is, exported data on S3 (for a single log group) are separated to several files and each file is unordered by timestamp.
I just wonder whether there is any way to get a single & sorted file through the export task or I should write a script to merge & sort the exported data.
Edit)
Exported files are separated like followings:
- 000000.gz
- 000001.gz
- 000002.gz
All files are unordered (timestamp of first 5 lines):
- 2021-12-03T11:13:34.909Z
- 2021-12-03T11:13:57.499Z
- 2021-12-03T11:14:34.909Z
- 2021-12-22T15:28:14.909Z // suddenly jumps
- 2021-12-22T15:31:14.909Z
Hello, thank you for reaching out!
At this time, it is the expected behavior for logs exported from CloudWatch to S3 using the CreateExportTask API to result in unordered log files.
As a result, it would be necessary to manually sort and combine the files using a script or, for example, you can use the below commands while using standard Linux tooling:
Example 1:
sort -k1 "000000 (1)" > 000000_sorted.txt
Example 2:
find . -exec zcat {} + | sed -r 's/^[0-9]+/\x0&/' | sort -z
Replace the '.' with the .gz file name that was downloaded from the S3 export and you can also pipe with another stdout command to another file so that you can save the sorted output
I can confirm that there is an open feature request to allow for ordered results when using CreateExportTask. While I am unable to comment on if/when this feature may get released, you can keep an eye on our What's New and Blog pages for any new feature announcements!
Relevant questions
Cloudwatch logs to S3 continuous export
asked 3 months agoIs it possible too Switch off or change the log level for CodeDeploy Logs?
asked 4 months agoThe logs aren't updating in cloudwatch log group for an ECS task
asked 13 days agoI would like to know easy way to start from the first line of log for the code-build execution (either in code-build console or cloudwatch-log console)
asked 2 months agoCan I use CloudWatch Logs to trigger a Lambda function?
Accepted Answerasked 2 years agoIs it possible to get the headers in HttpApi Gateway logs?
asked 12 days agoAre we able to export only parts of the Amazon CloudWatch logs to Amazon S3?
Accepted Answerasked 2 years agoHow to stream CloudFront real time logs to cloudwatch
asked 5 months agoCloudwatch Log Insights doesn't find logs from the first Lambda Invocation
asked 11 days agoIs it possible to get a single & sorted exported log data from CloudWatch Logs export task?
Accepted Answerasked 5 months ago
Thanks for the kind explanation! :D