cloudwatch agent log custom date time format

0

Hi I'm currently using aws cloudwatch agent for pushing custom log to aws.

Sample log line:

v:1,dt:2023-12-22T11:22:33+00:00,active:1,idle:34,totalRequest:363944,slow:76,traffic:1.5

Cloud watch config (/etc/awslogs/awslogs/conf)

[sample-log]
datetime_format = "dt:%Y-%m-%dT%H:%M:%S+00:00"
file = /path/to/log
log_group_name = log-group-name
log_stream_name = {instance_id}
retention_in_days = 90

My question are:

  1. The date format must be quoted, if not log group will not created (error in parsing ?). Why is it must be quoted ? is there a documentation about it ?
  2. Even when quoted, log group is created, but log entry still not recognizing timestamp in the log line, it is using log fetch time.

I found it very hard and confusing, many trial error and guesses, are there any detail documentation about datetime_format parsing ? is there a way to test an expression with a matching line (tool/code) ?

I even have tried using python class DateTimeParser from AWS parser.py source.

in ec2 server with aws log agent installed:
yum list installed | grep aws
repoquery -l aws-cli-plugin-cloudwatch-logs
copy file /usr/lib/python2.7/site-packages/cwlogs/parser.py
class DateTimeParser

The syntax parsed correctly for the expression vs log line, but when applied to aws log daemon, it wont run or wont recognize properly.

coolpie
posta 10 mesi fa986 visualizzazioni
1 Risposta
0

Hi, yes, there is a doc re. datetime_format for CloudWatch agent: see https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html and search for datetime_format.

Best,

Didier

profile pictureAWS
ESPERTO
con risposta 10 mesi fa
  • hi Didier, yes i've already read that doc, and also have spent many time googling, but so far there are no other detailed documentation about dateformat parsing. Thats why i have to run AWS python source code to verify it. (question updated: how to get the parser file)

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande