Create notification from logs

0

I would like to create a notification when a json file, that gets injected in my logs, gets updated.

We have a service that queries an external service and gets a response in json format. It queries this service every 5 minutes. Most of the time, the json is the same as the previous one, it only changes when it has an update. It is these updates that I would like to catch and trigger a notification.

I only have access to the logs where the json shows.

Would you have any suggestion on how to check that log and if the json has changed, then trigger a notification?

Miguel
已提问 1 个月前172 查看次数
1 回答
1

Even with. this limited amount of information, there may be more than one way to skin this particular cat.

One approach (1) is to feed the cloudwatch logs to a Lambda function, Kinesis Data Streams or Firehose through a log group-level subscription filter, let it go through every log message to find the aforementioned JSON response, canonicalize and compare against a last-saved version and determine if it has changed to trigger a notification. On first sight this feels rather expensive in terms of effort and resource utilization.

Assuming that the external service is HTTP or network-based, another approach (2) would be to insert a proxy between the external service and your consumer, and perform the change-detection logic locally. And here we're talking about both an HTTP proxy and the actual proxy microservice design pattern, mind you. This component doesn't need to perform any transformations in the content, just to parrot a request, wait for the response and forward it blindly, and then canonicalize it, compare and trigger the notification when applicable. We're talking about a few lines of python, nodejs or perl in a Lambda. Depending on the level of control you have on the consumer, you may just (a) change the address for that single endpoint, (b) define an HTTP_PROXY environment variable or (c) install an iptables transparent proxy (like squid and mitmproxy do). Please note that (2.a) and (2.b) are a one-afternoon project, but (2.c) might be less cost-effective than approach 1, and it's fair to say that in any case you are adding a moving part that depending on the circumstances can become another possible point of failure.

profile picture
已回答 10 天前
  • Thanks Javier for the few approaches which have triggered some other ideas.

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则