Subscription filter on comma-delimited (CSV) log

1

Hi there,

I'm trying to stream RDS Audit Logs from Cloudwatch Logs to Elasticsearch. It seems the filter pattern doesn't understand CSV out of the box and only looks for spaces. I can't figure out how to parse it properly.

20200612 23:16:23,ip-172-xx-xx-xx,username1,10.0.0.1,545598,57639790,QUERY,dbname,'SELECT * FROM users LIMIT 1',0

Using the Insights dashboard I can parse it fine using the code below, but this sintax doesn't work on the Filter pattern box.

parse '* *,*,*,*,*,*,*,*,*,*' as date, time, serverhost, username, host, connectionid, queryid, operation, database, object, retcode

I tried a few parse rules using the brackets notation (\[date, time, ...]) but that doesn't work either. Does anyone know how to feed CSV into the filter patterns? Or how to make the parse command work on the subscription filter?

질문됨 4년 전884회 조회
3개 답변
2

Hello,

Subscription Log Filters support only space delimited and JSON parsing. Also, be aware that Log Filter and Log Insights have different syntax.

However, I believe the best use of Subscription Log Filter is to filter log events, and not to parse them. Note that in your example, you attempt to retrieve every field. If your intent is to send all data in the log event to ElasticSearch, then you need not include a log filter at-all. Try clicking the ‘Test pattern’ button with an empty Subscription filter pattern to see this working.

Finally, when you create your log subscription to ElasticSearch, a Lambda function will be created on your behalf to handle the log event. If you need your log event parsed, say to conform to a specific schema for ElasticSearch, you may be better served parsing the log entry in the Lambda function, where you’ll have the full power of the programming language of your choice.

Best,
Kevin

AWS
답변함 4년 전
0

Any chance I can get some help on this Parser / Filter issue? :)

답변함 4년 전
0

Hi Kevin,

Thanks for the reply! I was under the impression a filter was mandatory to stream logs to ES. Definitely sending everything through is better, and adding login/parsing rules to Lambda also helps.

Thanks!

답변함 4년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인