Is it possible to get a single & sorted exported log data from CloudWatch Logs export task?

1

Currently I use the CreateExportTask API to backup my log data.

The problem is, exported data on S3 (for a single log group) are separated to several files and each file is unordered by timestamp.

I just wonder whether there is any way to get a single & sorted file through the export task or I should write a script to merge & sort the exported data.

Edit)

Exported files are separated like followings:

  • 000000.gz
  • 000001.gz
  • 000002.gz

All files are unordered (timestamp of first 5 lines):

  1. 2021-12-03T11:13:34.909Z
  2. 2021-12-03T11:13:57.499Z
  3. 2021-12-03T11:14:34.909Z
  4. 2021-12-22T15:28:14.909Z // suddenly jumps
  5. 2021-12-22T15:31:14.909Z
1 réponse
2
Réponse acceptée

Hello, thank you for reaching out!

At this time, it is the expected behavior for logs exported from CloudWatch to S3 using the CreateExportTask API to result in unordered log files.

As a result, it would be necessary to manually sort and combine the files using a script or, for example, you can use the below commands while using standard Linux tooling:

Example 1:

sort -k1 "000000 (1)" > 000000_sorted.txt

Example 2:

find . -exec zcat {} + | sed -r 's/^[0-9]+/\x0&/' | sort -z

Replace the '.' with the .gz file name that was downloaded from the S3 export and you can also pipe with another stdout command to another file so that you can save the sorted output

I can confirm that there is an open feature request to allow for ordered results when using CreateExportTask. While I am unable to comment on if/when this feature may get released, you can keep an eye on our What's New and Blog pages for any new feature announcements!

AWS
INGÉNIEUR EN ASSISTANCE TECHNIQUE
répondu il y a 2 ans
AWS
EXPERT
vérifié il y a 2 ans

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions