Is it possible to get a single & sorted exported log data from CloudWatch Logs export task?

1

Currently I use the CreateExportTask API to backup my log data.

The problem is, exported data on S3 (for a single log group) are separated to several files and each file is unordered by timestamp.

I just wonder whether there is any way to get a single & sorted file through the export task or I should write a script to merge & sort the exported data.

Edit)

Exported files are separated like followings:

  • 000000.gz
  • 000001.gz
  • 000002.gz

All files are unordered (timestamp of first 5 lines):

  1. 2021-12-03T11:13:34.909Z
  2. 2021-12-03T11:13:57.499Z
  3. 2021-12-03T11:14:34.909Z
  4. 2021-12-22T15:28:14.909Z // suddenly jumps
  5. 2021-12-22T15:31:14.909Z
1 Answer
2
Accepted Answer

Hello, thank you for reaching out!

At this time, it is the expected behavior for logs exported from CloudWatch to S3 using the CreateExportTask API to result in unordered log files.

As a result, it would be necessary to manually sort and combine the files using a script or, for example, you can use the below commands while using standard Linux tooling:

Example 1:

sort -k1 "000000 (1)" > 000000_sorted.txt

Example 2:

find . -exec zcat {} + | sed -r 's/^[0-9]+/\x0&/' | sort -z

Replace the '.' with the .gz file name that was downloaded from the S3 export and you can also pipe with another stdout command to another file so that you can save the sorted output

I can confirm that there is an open feature request to allow for ordered results when using CreateExportTask. While I am unable to comment on if/when this feature may get released, you can keep an eye on our What's New and Blog pages for any new feature announcements!

AWS
SUPPORT ENGINEER
answered 2 years ago
AWS
EXPERT
reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions