CloudWatch logs are not reaching Splunk for the new lambda with AWS Firehose/Lambda integration

0

hi, I need help with understanding Firehose integration with CW log streams for Splunk logging. I already looked at AWS documentation. I can see that CWToSplunk subscription filter with the Firehose delivery stream already setup in our dev account for the log group but the log streams are not reaching Splunk. Just wanted to understand what config is possibly missing?!!

This CW-Firehose-Splunk setup is supposed to be working for previous Lambdas. I created a new java based Lambda and when it is invoked, I am able to see that application log written by Log back is reaching to the CouldWatch log group, but never showed up in Splunk. I also verified that existing CWToSplunk delivery stream has Splunk as the correct destination and tranformation Lambda is enabled. What would be different for a new lambda if previous Lambda in the same account with the same setup is already working with firehose integration and is reaching splunk?? Thanks.

1 Answer
0

Hi, and thanks for reaching out!

There may be one of a couple things happening here:

  1. You can verify if the subscription filter is failing or succeeding in sending log events to the subscription filter destination by checking the CloudWatch Logs metric for the log group. If you see DeliveryErrors metric data for the Lambda's log group, it indicates that CloudWatch Logs is attempting to send data to the destination, but failing to do so, usually due to a permissions issue.

  2. Double check that the subscription filter pattern matches against log events you wish to send to the subscription destination.

  3. Ensure that the IAM role being used for the subscription filter has appropriate trust policy and permissions policy statements to allow the "logs.<region>.amazonaws.com" service to assume the role and send data to the destination Kinesis Firehose (as seen in Step 8 here).

  4. Ensure that the Firehose role has an appropriate trust policy and permissions policy to allow the firehose service to assume the role, and for it to perform appropriate actions in sending to Splunk

  5. If utilizing the Amazon Kinesis Firehose Splunk Add-on, you can verify the Splunk configuration items per the Splunk documentation.

If these all seem in order, I would recommend opening a case in the AWS Support Center and working with an available engineer to provide further visibility on your resources and track down any other issues with the integration.

AWS
SUPPORT ENGINEER
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions