cloudwatch agent log custom date time format

0

Hi I'm currently using aws cloudwatch agent for pushing custom log to aws.

Sample log line:

v:1,dt:2023-12-22T11:22:33+00:00,active:1,idle:34,totalRequest:363944,slow:76,traffic:1.5

Cloud watch config (/etc/awslogs/awslogs/conf)

[sample-log]
datetime_format = "dt:%Y-%m-%dT%H:%M:%S+00:00"
file = /path/to/log
log_group_name = log-group-name
log_stream_name = {instance_id}
retention_in_days = 90

My question are:

  1. The date format must be quoted, if not log group will not created (error in parsing ?). Why is it must be quoted ? is there a documentation about it ?
  2. Even when quoted, log group is created, but log entry still not recognizing timestamp in the log line, it is using log fetch time.

I found it very hard and confusing, many trial error and guesses, are there any detail documentation about datetime_format parsing ? is there a way to test an expression with a matching line (tool/code) ?

I even have tried using python class DateTimeParser from AWS parser.py source.

in ec2 server with aws log agent installed:
yum list installed | grep aws
repoquery -l aws-cli-plugin-cloudwatch-logs
copy file /usr/lib/python2.7/site-packages/cwlogs/parser.py
class DateTimeParser

The syntax parsed correctly for the expression vs log line, but when applied to aws log daemon, it wont run or wont recognize properly.

coolpie
asked 9 months ago949 views
1 Answer
0

Hi, yes, there is a doc re. datetime_format for CloudWatch agent: see https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html and search for datetime_format.

Best,

Didier

profile pictureAWS
EXPERT
answered 9 months ago
  • hi Didier, yes i've already read that doc, and also have spent many time googling, but so far there are no other detailed documentation about dateformat parsing. Thats why i have to run AWS python source code to verify it. (question updated: how to get the parser file)

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions