Send CloudWatch logs to Splunk

Question

Hi, architectural question here. 
Goal is to move logs from cloudwatch logs to an on premises Splunk, reliably.

I see different options, with trade-off:
1. add a lambda subscription filter to the log group and leverage a lambda Splunk blueprint function which is able to push logs. Simple, but risk of throttling if huge amount of logs is sent
2. Combination of EventBridge scheduler + lambda to move data to s3, and then via fan-out pattern (sns,SQS) to the Splunk lambda logging function. This is more reliable but getting complex and not a big fun of schedules.
3. Same as above, but using kinesis to move data to s3, and then fanning out to Splunk lambda as before. Also a constraint here is that kinesis may not usable for certain reason.
4. Cloudwatch subscription filter with a lambda that push data to SNS. Topic is consumed either directly via Splunk lambda, or via SQS which then the Splunk lambda listen too. Risk about throttling and slightly complex architecture.
5. Need to check feasibility, but was looking for an EventBridge tule, which may listen to log groups and move logs to SNS, and then to Splunk, but haven’t confirmed this yet.

Any other alternatives? Thanks

Answer

Send CloudWatch logs to Kinesis Firehose https://repost.aws/knowledge-center/cloudwatch-logs-stream-to-kinesis

Kinesis Firehose streams can be sent directly to Splunk for ingestion https://aws.amazon.com/kinesis/data-firehose/splunk/ https://docs.splunk.com/Documentation/AddOns/released/Firehose/ConfigureFirehose

Answer

Why not try Splunk Addon for AWS as outlined here - https://docs.splunk.com/Documentation/AddOns/released/AWS/CloudWatchLogs

Answer

Is this to batch move or stream real time as you mention move but then with subscription filters I believe they will only monitor for new events from that point of configuration and not historical events? (Need to confirm)

You could export logs directly to S3 from the console and then import from there. When I have done this in the past, I believe I could only export log groups 1 or 2 at a time.

Answer

I would recommend considering using Amazon Kinesis Data Firehose to reliably deliver logs from CloudWatch Logs to Splunk.

Some key advantages of this approach:

* Kinesis Data Firehose can automatically deliver log data from CloudWatch Logs to Splunk with minimal code required. It handles log aggregation, compression and transport securely at a large scale.
* Firehose delivers log data reliably to Splunk with options for data transformation along the way if needed. It can also handle high volumes of log data from CloudWatch Logs.
* This avoids the need to build out and manage your own log delivery infrastructure using Lambda, SNS/SQS etc. which comes with additional operational overhead.
* Splunk has documentation on how to configure Firehose for log delivery directly to Splunk for ingestion.

To get started, you can create a Firehose delivery stream that sources data from a CloudWatch Logs group and delivers to your Splunk endpoint. The AWS documentation provides steps to set this up. Let me know if you have any other questions!

Send CloudWatch logs to Splunk

Relevant content