JSON logs not seen as JSON by Subscription filter?

Question

I'm trying to get our logs from cloudwatch into Kibana, but I've run into an unexpected problem. Our logs are JSON formatted, and show up as such in cloudwatch. When I go to create an Elasticsearch subscription filter, and choose the JSON format testing the filter pattern on the data from this log group just matches the whole json object as a string it puts under the message field. If I try to add a pattern, it seems to split the string on random delimiters (: , and space) and that wouldn't help anyway since the logs don't all have the same json fields.   
  
Am I approaching this wrong?   
How do I get json data from cloudwatch into elastic search with the fields being the same on either end?

percosab · Answer

I figured this out. The JSON format was working, but all my log groups were going to the same index, once we fixed that they started showing up with all their fields in ES. I did end up adding a pattern, { $.written_ts > 500 }, just to sort out the actual JSON objects from the occaisional print() statement that I still need to find and remove.

percosab

respondido hace 6 años

JSON logs not seen as JSON by Subscription filter?

Añade tu respuesta

Contenido relevante