Subscription filter on comma-delimited (CSV) log

Question

Hi there,  
  
I'm trying to stream RDS Audit Logs from Cloudwatch Logs to Elasticsearch. It seems the filter pattern doesn't understand CSV out of the box and only looks for spaces. I can't figure out how to parse it properly.

```
20200612 23:16:23,ip-172-xx-xx-xx,username1,10.0.0.1,545598,57639790,QUERY,dbname,'SELECT * FROM users LIMIT 1',0
```

Using the Insights dashboard I can parse it fine using the code below, but this sintax doesn't work on the Filter pattern box.

```
parse '* *,*,*,*,*,*,*,*,*,*' as date, time, serverhost, username, host, connectionid, queryid, operation, database, object, retcode
```

I tried a few parse rules using the brackets notation ($$date, time, ...$$) but that doesn't work either. Does anyone know how to feed CSV into the filter patterns? Or how to make the **parse** command work on the subscription filter?

Answer

Hello,  
  
Subscription Log Filters support only space delimited and JSON parsing. Also, be aware that Log Filter and Log Insights have different syntax.  
  
However, I believe the best use of Subscription Log Filter is to filter log events, and not to parse them. Note that in your example, you attempt to retrieve every field. If your intent is to send all data in the log event to ElasticSearch, then you need not include a log filter at-all. Try clicking the ‘Test pattern’ button with an empty Subscription filter pattern to see this working.  
  
Finally, when you create your log subscription to ElasticSearch, a Lambda function will be created on your behalf to handle the log event. If you need your log event parsed, say to conform to a specific schema for ElasticSearch, you may be better served parsing the log entry in the Lambda function, where you’ll have the full power of the programming language of your choice.  
  
Best,  
Kevin

Answer

Hi Kevin,  
  
Thanks for the reply! I was under the impression a filter was mandatory to stream logs to ES. Definitely sending everything through is better, and adding login/parsing rules to Lambda also helps.  
  
Thanks!

Answer

Any chance I can get some help on this Parser / Filter issue? :)

Subscription filter on comma-delimited (CSV) log

Relevant content