Is it possible to get a single & sorted exported log data from CloudWatch Logs export task?

1

Currently I use the CreateExportTask API to backup my log data.

The problem is, exported data on S3 (for a single log group) are separated to several files and each file is unordered by timestamp.

I just wonder whether there is any way to get a single & sorted file through the export task or I should write a script to merge & sort the exported data.

Edit)

Exported files are separated like followings:

  • 000000.gz
  • 000001.gz
  • 000002.gz

All files are unordered (timestamp of first 5 lines):

  1. 2021-12-03T11:13:34.909Z
  2. 2021-12-03T11:13:57.499Z
  3. 2021-12-03T11:14:34.909Z
  4. 2021-12-22T15:28:14.909Z // suddenly jumps
  5. 2021-12-22T15:31:14.909Z
1 回答
2
已接受的回答

Hello, thank you for reaching out!

At this time, it is the expected behavior for logs exported from CloudWatch to S3 using the CreateExportTask API to result in unordered log files.

As a result, it would be necessary to manually sort and combine the files using a script or, for example, you can use the below commands while using standard Linux tooling:

Example 1:

sort -k1 "000000 (1)" > 000000_sorted.txt

Example 2:

find . -exec zcat {} + | sed -r 's/^[0-9]+/\x0&/' | sort -z

Replace the '.' with the .gz file name that was downloaded from the S3 export and you can also pipe with another stdout command to another file so that you can save the sorted output

I can confirm that there is an open feature request to allow for ordered results when using CreateExportTask. While I am unable to comment on if/when this feature may get released, you can keep an eye on our What's New and Blog pages for any new feature announcements!

AWS
支持工程师
已回答 2 年前
AWS
专家
已审核 2 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则