Is it possible to get a single & sorted exported log data from CloudWatch Logs export task?

1

Currently I use the CreateExportTask API to backup my log data.

The problem is, exported data on S3 (for a single log group) are separated to several files and each file is unordered by timestamp.

I just wonder whether there is any way to get a single & sorted file through the export task or I should write a script to merge & sort the exported data.

Edit)

Exported files are separated like followings:

  • 000000.gz
  • 000001.gz
  • 000002.gz

All files are unordered (timestamp of first 5 lines):

  1. 2021-12-03T11:13:34.909Z
  2. 2021-12-03T11:13:57.499Z
  3. 2021-12-03T11:14:34.909Z
  4. 2021-12-22T15:28:14.909Z // suddenly jumps
  5. 2021-12-22T15:31:14.909Z
1개 답변
2
수락된 답변

Hello, thank you for reaching out!

At this time, it is the expected behavior for logs exported from CloudWatch to S3 using the CreateExportTask API to result in unordered log files.

As a result, it would be necessary to manually sort and combine the files using a script or, for example, you can use the below commands while using standard Linux tooling:

Example 1:

sort -k1 "000000 (1)" > 000000_sorted.txt

Example 2:

find . -exec zcat {} + | sed -r 's/^[0-9]+/\x0&/' | sort -z

Replace the '.' with the .gz file name that was downloaded from the S3 export and you can also pipe with another stdout command to another file so that you can save the sorted output

I can confirm that there is an open feature request to allow for ordered results when using CreateExportTask. While I am unable to comment on if/when this feature may get released, you can keep an eye on our What's New and Blog pages for any new feature announcements!

AWS
지원 엔지니어
답변함 2년 전
AWS
전문가
검토됨 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠