Cloudwatch Log Insights - Line Chart Issues

Question

Hello :wave:

We are trying to use Cloudwatch Logs together with Cloudwatch Log Insights to Query through application's logs and visualise the API usage by different consumers. A single log consists of fields such as:

- consumer (api user)
- controller (base endpoint on which the consumer made a request)
- action (exact action on which someone sent a request)

Sample entry could look like this (obviously it truncated for the sake of explaining the problem)

```
{
  consumer: 'test-app-1',
  controller: 'api/test-controller',
  action: 'test-action
}
```

Now, I would like to find a way in which I can group this in a line chart, for example requests made to different controllers by the consumers over time or exact action requests by consumers over time.

I am not sure how to write the query, if I am to be honest, as 
```
filter @logStream = 'someLogStream'
 | fields @timestamp, @message
 | stats count() by consumer, controller, action
```
does not generate a line chart over time, and adding
```
filter @logStream = 'someLogStream'
 | fields @timestamp, @message
 | stats count() by consumer, controller, action, bin(15m)
```
does not change the error message on visualisation screen, which states that:
>The data is not suitable for a line chart.
Try a bar chart, or group your result by bin function.

Can anyone advise on what should I do? For starters, requests over time per consumer would be a nice metric and I believe that once we get to that point, we will be able to modify it per action / per controller. I think that I lack proficiency in the Insights syntax to achieve what I want.

Thanks!

Answer

Thank you for this question. You are right about a lot of things here. Below I have explained a few pointers, that can help in writing your queries.

The following statement needs to be changed, for proper use of the `bin()` function:

```
stats count() by consumer, controller, action, bin(15m)
```

* In the above statement, you might have observed an error (on hovering your mouse on the underlined red-lines in your query on console) stating that `bin()` is not valid in this position - because, EOF or other applicable keywords are expected after `by` in the query. For example `stats count(action = 'test-action') by bin(15m)` would be valid.
* `count()` expects parameters i.e. either `count()` or `count(fieldName: LogField)`. You can also use `*` as your parameter like this - `count(*)`. Read more [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html#CWL_Insights_Aggregation_Functions).

Coming to your use case to count number of requests for a specific *consumer* - for example: to count the number of log events for consumer "test-app-1", change your query to something like the following:
```
 filter @logStream = 'someLogStream' and consumer = 'test-app-1'
 | fields @timestamp, @message
 | stats count(*) by bin(15m)
```
Notice that I have used the `filter` statement in order to only fetch the results for a specific consumer, feel free to experiment with such options for other filtering that you might want to perform.

I recommend reading the AWS documentation titled ["CloudWatch Logs Insights query syntax"](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html). Specifically, for this question - the sections *"Matches and regular expressions in the filter command"* [[1]](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html#CWL_QuerySyntax-regex) & *"Supported operations and functions"* [[2]](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html#CWL_QuerySyntax-operations-functions) can be helpful in getting more clarity & for further learning.

I hope this answer helps in starting with your analysis using Log Insights queries. Feel free to comment on any clarification needed.

***Happy learning!***

Cloudwatch Log Insights - Line Chart Issues

Relevant content