Is it possible to get a single & sorted exported log data from CloudWatch Logs export task?

1

Currently I use the CreateExportTask API to backup my log data.

The problem is, exported data on S3 (for a single log group) are separated to several files and each file is unordered by timestamp.

I just wonder whether there is any way to get a single & sorted file through the export task or I should write a script to merge & sort the exported data.

Edit)

Exported files are separated like followings:

  • 000000.gz
  • 000001.gz
  • 000002.gz

All files are unordered (timestamp of first 5 lines):

  1. 2021-12-03T11:13:34.909Z
  2. 2021-12-03T11:13:57.499Z
  3. 2021-12-03T11:14:34.909Z
  4. 2021-12-22T15:28:14.909Z // suddenly jumps
  5. 2021-12-22T15:31:14.909Z
1 Antwort
2
Akzeptierte Antwort

Hello, thank you for reaching out!

At this time, it is the expected behavior for logs exported from CloudWatch to S3 using the CreateExportTask API to result in unordered log files.

As a result, it would be necessary to manually sort and combine the files using a script or, for example, you can use the below commands while using standard Linux tooling:

Example 1:

sort -k1 "000000 (1)" > 000000_sorted.txt

Example 2:

find . -exec zcat {} + | sed -r 's/^[0-9]+/\x0&/' | sort -z

Replace the '.' with the .gz file name that was downloaded from the S3 export and you can also pipe with another stdout command to another file so that you can save the sorted output

I can confirm that there is an open feature request to allow for ordered results when using CreateExportTask. While I am unable to comment on if/when this feature may get released, you can keep an eye on our What's New and Blog pages for any new feature announcements!

AWS
SUPPORT-TECHNIKER
beantwortet vor 2 Jahren
AWS
EXPERTE
überprüft vor 2 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen