CloudWatch logs are not reaching Splunk for the new lambda with AWS Firehose/Lambda integration
hi, I need help with understanding Firehose integration with CW log streams for Splunk logging. I already looked at AWS documentation. I can see that CWToSplunk subscription filter with the Firehose delivery stream already setup in our dev account for the log group but the log streams are not reaching Splunk. Just wanted to understand what config is possibly missing?!!
This CW-Firehose-Splunk setup is supposed to be working for previous Lambdas. I created a new java based Lambda and when it is invoked, I am able to see that application log written by Log back is reaching to the CouldWatch log group, but never showed up in Splunk. I also verified that existing CWToSplunk delivery stream has Splunk as the correct destination and tranformation Lambda is enabled. What would be different for a new lambda if previous Lambda in the same account with the same setup is already working with firehose integration and is reaching splunk?? Thanks.
Hi, and thanks for reaching out!
There may be one of a couple things happening here:
You can verify if the subscription filter is failing or succeeding in sending log events to the subscription filter destination by checking the CloudWatch Logs metric for the log group. If you see DeliveryErrors metric data for the Lambda's log group, it indicates that CloudWatch Logs is attempting to send data to the destination, but failing to do so, usually due to a permissions issue.
Double check that the subscription filter pattern matches against log events you wish to send to the subscription destination.
Ensure that the IAM role being used for the subscription filter has appropriate trust policy and permissions policy statements to allow the "logs.<region>.amazonaws.com" service to assume the role and send data to the destination Kinesis Firehose (as seen in Step 8 here).
Ensure that the Firehose role has an appropriate trust policy and permissions policy to allow the firehose service to assume the role, and for it to perform appropriate actions in sending to Splunk
If utilizing the Amazon Kinesis Firehose Splunk Add-on, you can verify the Splunk configuration items per the Splunk documentation.
If these all seem in order, I would recommend opening a case in the AWS Support Center and working with an available engineer to provide further visibility on your resources and track down any other issues with the integration.
CloudWatch logs are not reaching Splunk for the new lambda with AWS Firehose/Lambda integrationasked a month ago
AWS Lambda Application with Integration and Production EnvironmentsAccepted Answerasked 4 months ago
Do I need to create two kinesis delivery streams to send messages under two different paths in the same s3 bucket?asked a month ago
I want consolidated application logs running on AWS ECS with microservices architectureasked a month ago
Are there any best practices for sending logs from ECS on EC2, ECS on Fargate and other AWS services such as API GW, load balancers (and more AWS services) to Splunk?asked 3 months ago
AWS CloudWatch metrics to OpenSearchasked 3 months ago
Binary uploads to API Gateway Proxy with Lambda IntegrationAccepted Answerasked 5 years ago
API Gateway access log still get requests blocked by WAFasked 3 months ago
Why set X-Amz-Target header for EventBridge integration?asked 2 years ago
Logging with CloudWatch vs. ElasticSearch/KibanaAccepted Answerasked 5 years ago