How do I push VPC flow logs to Splunk using Amazon Kinesis Firehose?

6 minute read

I want to improve the integration of my AWS data with Splunk when pushing data from AWS sources to a Splunk cluster.

Resolution

Note: Kinesis Data Firehose doesn't support Application Load Balancers or Network Load Balancers. Instead, use a Classic Load Balancer. Make sure to allow duration-based sticky sessions with cookie expiration deactivated. For more information about troubleshooting delivery stream issues with Splunk endpoints, see Data not delivered to Splunk.

Prerequisites

Set up a reachable Splunk HEC HTTP Event Collector (HEC) instance. For more information, see Splunk Enterprise on AWS and Set up and use HTTP Event Collector in Splunk Web on the Splunk website.
Configure the Splunk data inputs with the HEC on Splunk Web.
Create a new HEC endpoint with a new token for configuring the data stream. For more information, see Set up and use HTTP Event Collector in Splunk Web (from the Splunk website). Make sure to choose the option Enable indexer acknowledgement.
Install the Splunk add-on for Amazon Kinesis Data Firehose from the Splunkbase website.

Create a Kinesis Data Firehose delivery stream

1. Create your delivery stream. For Source, choose Direct PUT.

2. For Destination, choose Splunk.

3. Enter a delivery stream name.

Configure record transformation with AWS Lambda

1. Configure record transformation.Note: Under Transform source records with AWS Lambda, choose Enabled for Record transformation. CloudWatch sends the logs as compressed .gzip files, and Amazon Kinesis must extract these files before you can use them.

2. For Lambda function, choose Create new.

3. On the Choose Lambda blueprint window, for Lambda blueprint, choose Kinesis Firehose CloudWatch Logs Processor.

4. Choose the new tab that opens in your browser to create the new Lambda function:
For Name, enter a name for the Lambda function.
For Role, choose Create a custom role.

5. Choose the new tab that opens in your browser to create a new AWS Identity and Access Management (IAM) role. For Role Name, make sure that the name is lambda_basic_execution.

6. Choose Allow to create the role, and then return to the Lambda function configuration page.

7. Choose Create function.

8. To prevent the function from timing out, increase the Timeout to 1 minute from the default 3 seconds.

9. Choose Save.

Finalize the Kinesis Data Firehose delivery stream

1. Open the Amazon Kinesis console.

2. In the navigation pane, choose Data Firehose.

3. For your delivery stream, choose Lambda function, and then from the dropdown, choose the name of your new AWS Lambda function.

4. For Destination Settings, enter the Splunk HEC details, including the Splunk HEC endpoint that you created.
Note: You must terminate the Splunk HEC endpoint with a valid SSL certificate. Use the matching DNS hostname to connect to your HEC endpoint. The format for the cluster endpoint is https://YOUR-ENDPOINT.splunk.com:8088.

5. For Splunk endpoint type, choose Raw endpoint, and then enter the authentication token.

6. Choose Next.

7. (Optional) Choose an existing Amazon Simple Storage Service (Amazon S3) bucket or create a new one to back up failed events or all events. Make sure to configure Amazon S3 related settings, such as buffer conditions, compression and encryption settings, and error logging options in the delivery stream wizard.

8. Under IAM role, choose Create New.

9. In the tab that opens, enter a Role name, and then choose Allow.

10. Choose Next.

11. Choose Create delivery stream.

Configure Amazon VPC Flow Logs

Note: If you have an Amazon VPC flow log, then proceed to the next section, Create an Amazon CloudWatch logs subscription. If you don't have a VPC flow log, then complete the following steps:

1. Open the CloudWatch console.

2. In the navigation pane, choose Logs.

3. For Actions, choose Create log group.

4. Enter a Log Group Name.

5. Choose Create log group.

6. Open the Amazon VPC console.

7. In the navigation pane under Virtual Private Cloud, choose Your VPCs.

8. In the content pane, select your VPC.

9. Choose the Flow logs view.

10. Choose Create flow log:

For Filter, choose All.
For Destination log group, choose the log group you just created.
For IAM role, choose an IAM role that allows your Amazon VPC to publish logs to CloudWatch. If you don't have an appropriate IAM role, then under IAM role, choose Set Up Permissions, and then choose Create a new IAM role. Keep the default settings.
Choose Allow to create and associate the role VPCFlowLogs with the destination log group.

11. Choose Create.

12. Establish a real-time feed from the log group to your delivery stream:

For AWS Lambda instructions, see Accessing Amazon CloudWatch Logs for AWS Lambda.
For Amazon OpenSearch Service instructions, see Streaming CloudWatch Logs data to Amazon OpenSearch Service.
For Kinesis Data Firehose, complete the steps in the Create a CloudWatch Logs subscription section to create a CloudWatch Logs subscription.

Note: If you receive errors when running AWS CLI commands, confirm that you're running a recent version of the AWS CLI.

Create an Amazon CloudWatch Logs subscription

1. Grant access to CloudWatch to publish your Kinesis Data Firehose stream with the correct role permissions.

2. Open AWS CLI.

3. Use the following JSON file to create your trust policy (such as TrustPolicyforCWLToFireHose.json). Replace YOUR-RESOURCE-REGION with your resource's AWS Region.

{
  "Statement": {
    "Effect": "Allow",
    "Principal": { "Service": "logs.YOUR-RESOURCE-REGION.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }
}

4. Run the following command to create the role with permissions from the trust policy:

$ aws iam create-role --role-name CWLtoKinesisFirehoseRole --assume-role-policy-document file://TrustPolicyForCWLToFireHose.json

5. Use the following JSON file to create your IAM policy (such as PermissionPolicyForCWLToFireHose.json). Replace YOUR-AWS-ACCT-NUM with your AWS account number, YOUR-RESOURCE-REGION with your resource's Region, and FirehoseSplunkDeliveryStream with your stream's name.

{
    "Statement":[
      {
        "Effect":"Allow",
        "Action":["firehose:PutRecord"],
        "Resource":["arn:aws:firehose:YOUR-RESOURCE-REGION:YOUR-AWS-ACCT-NUM:deliverystream/FirehoseSplunkDeliveryStream"]
      },
      {
        "Effect":"Allow",
        "Action":["iam:PassRole"],
        "Resource":["arn:aws:iam::YOUR-AWS-ACCT-NUM:role/CWLtoKinesisFirehoseRole"]
      }
    ]
}

6. Run the following command to attach the IAM policy to the role:

$ aws iam put-role-policy 
    --role-name CWLtoKinesisFirehoseRole 
    --policy-name Permissions-Policy-For-CWL 
    --policy-document file://PermissionPolicyForCWLToFireHose.json

7. Run the following command to create a subscription filter. Replace YOUR-AWS-ACCT-NUM with your AWS account number, YOUR-RESOURCE-REGION with your resource's Region, and FirehoseSplunkDeliveryStream with your stream's name.

$ aws logs put-subscription-filter 
   --log-group-name " /vpc/flowlog/FirehoseSplunk" 
   --filter-name "Destination" 
   --filter-pattern "" 
   --destination-arn "arn:aws:firehose:YOUR-RESOURCE-REGION:YOUR-AWS-ACCT-NUM:deliverystream/FirehoseSplunkDeliveryStream" 
   --role-arn "arn:aws:iam::YOUR-AWS-ACCT-NUM:role/CWLtoKinesisFirehoseRole"

For more information on subscription filters, see Example 3: subscription filters with Amazon Kinesis Data Firehose.