cloudwatch agent log custom date time format

0

Hi I'm currently using aws cloudwatch agent for pushing custom log to aws.

Sample log line:

v:1,dt:2023-12-22T11:22:33+00:00,active:1,idle:34,totalRequest:363944,slow:76,traffic:1.5

Cloud watch config (/etc/awslogs/awslogs/conf)

[sample-log]
datetime_format = "dt:%Y-%m-%dT%H:%M:%S+00:00"
file = /path/to/log
log_group_name = log-group-name
log_stream_name = {instance_id}
retention_in_days = 90

My question are:

  1. The date format must be quoted, if not log group will not created (error in parsing ?). Why is it must be quoted ? is there a documentation about it ?
  2. Even when quoted, log group is created, but log entry still not recognizing timestamp in the log line, it is using log fetch time.

I found it very hard and confusing, many trial error and guesses, are there any detail documentation about datetime_format parsing ? is there a way to test an expression with a matching line (tool/code) ?

I even have tried using python class DateTimeParser from AWS parser.py source.

in ec2 server with aws log agent installed:
yum list installed | grep aws
repoquery -l aws-cli-plugin-cloudwatch-logs
copy file /usr/lib/python2.7/site-packages/cwlogs/parser.py
class DateTimeParser

The syntax parsed correctly for the expression vs log line, but when applied to aws log daemon, it wont run or wont recognize properly.

coolpie
asked 10 months ago1039 views
1 Answer
0

Hi, yes, there is a doc re. datetime_format for CloudWatch agent: see https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html and search for datetime_format.

Best,

Didier

profile pictureAWS
EXPERT
answered 10 months ago
  • hi Didier, yes i've already read that doc, and also have spent many time googling, but so far there are no other detailed documentation about dateformat parsing. Thats why i have to run AWS python source code to verify it. (question updated: how to get the parser file)

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions