How do I create, configure, and troubleshoot a subscription filter to Kinesis through the CloudWatch console?

Resolution

You can use a subscription filter to send CloudWatch logs in near real-time. You can send logs to the same account, to cross-account Kinesis, or to Amazon Data Firehose destinations. The CloudWatch Logs console supports the destination and setup configuration. For information, see Filter pattern syntax for metric filters, subscription filters, filter log events, and Live Tail.

Subscription configuration for a Kinesis data stream in the same account

Note: The AWS Region for the CloudWatch log group and the Kinesis destination must be the same.

Before you create your subscription, complete the following actions:

• Create a Kinesis stream, if you don't already have one.
• Create an AWS Identity and Access Management (IAM) role that has the appropriate trust policy and permissions.

To create a custom IAM role and role policy, complete the following steps:

1. Open the IAM console with the user that has administrator permissions.

2. From the navigation pane, choose Policies.

3. Choose Create policy.

4. Enter the following role permissions policy document into the JSON tab.
   Replace REGION, ACCOUNT_ID, AND STREAM_NAME with your own details.
   Note: The following permission policy is specific to Kinesis streams. For more information on permissions for Firehose, see Controlling access with Amazon Data Firehose.

   {  "Statement": {
       "Effect": "Allow",
       "Action": "kinesis:PutRecord",
       "Resource": "arn:aws:kinesis:REGION:ACCOUNT_ID:stream/STREAM_NAME"
     }
   }

5. Choose Next, and then choose Create policy.

6. From navigation pane, choose Roles, and then choose Create role.

7. Choose the Custom trust policy role type.

8. In the Custom trust policy section, enter the custom trust policy for the role.
   The following is an example trust policy. Replace REGION and ACCOUNT_ID with your own details.

   {  "Statement": {
       "Effect": "Allow",
       "Principal": {
         "Service": "logs.region.amazonaws.com"
       },
       "Action": "sts:AssumeRole",
       "Condition": {
         "StringLike": {
           "aws:SourceArn": "arn:aws:logs:REGION:ACCOUNT_ID:*"
         }
       }
     }
   }

9. Choose Next.

10. Select the custom IAM policy that you created in step 4.

11. Choose Next, and then Create a Role.

Create a subscription filter

After you create the Kinesis stream and the IAM role, create the subscription filter:

1. Open the CloudWatch console.

2. Choose Log Group.

3. Choose Action, and then Subscription Filters.

4. To configure the destination, choose Create Amazon Data Firehose subscription filter.

5. Choose Current Account.

6. Select your Kinesis data stream from the dropdown list.

7. Select the IAM role that you created.

8. Choose the Distribution method:

   • By Log Stream: This verifies that downstream consumers can aggregate log events by log stream, but the method is less efficient. This method might also incur higher streaming costs because it requires more shards.
   • Random: This distributes the load across Kinesis stream shards, but downstream consumers can't aggregate log events by log stream.

9. Configure Log format and filters:

   Select the log format. The format can be Amazon VPC Flow Logs, AWS CloudTrail, or AWS Lambda, for logs that are published by Amazon VPC, CloudTrail, or Lambda. You can also choose JSON, Space delimited, or Other, depending on your incoming log events.
   Define the filter pattern in the Subscription filter pattern section.
   Enter a name for your subscription filter.

10. Verify your pattern with the existing log event data.

11. Choose Start Streaming.

12. (Optional) To verify that your data stream works, validate the flow of log events.

Configuration for a cross-account Kinesis data stream destination

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

You can deliver CloudWatch Logs events to Kinesis data streams in different AWS accounts and Regions. To do this, set up cross-account log data sharing with subscriptions and specify the Region.

Create a destination data stream in the recipient account

Create a destination data stream in Kinesis in the data recipient's account with an IAM role and trust policy.

Create a subscription filter in the source account

Complete the following steps:

1. Open the CloudWatch console.
2. Choose Log Group.
3. Select the log group.
4. Choose Action, and then choose Subscription Filters.
5. To select the destination, choose Create, and then select Create Amazon Data Firehose subscription filter.
6. Choose Another Account.
7. For a cross-account Kinesis or Kinesis Data Firehose destination, provide the Destination ARN.
8. Choose the Distribution method.
9. Configure Log format and filters.
10. Verify your pattern with the existing log event data.
11. Choose Start Streaming.
12. (Optional) To verify that your data stream works, validate the flow log events.

Troubleshooting

• Make sure that your Kinesis stream is in the Active state. You can view the stream on the Kinesis console or use the DescribeStream API call.
• Verify that the CloudWatch log group and Kinesis data stream Regions are the same.
• Make sure that there is an IAM role that has trust permissions for logs.yourregion.amazonaws.com and allows the permission kinesis:putrecords.
• Verify that the Regions and resources in the IAM policy are correct.
• Make sure that you didn't select Create Kinesis subscription filter when you configure a subscription filter for Kinesis data stream.
• After you start streaming, check the subscription filter metrics to confirm that the filter pattern is valid and matches the incoming log events.
  Review the following metrics:
  ForwardedBytes: The volume of log events in compressed bytes forwarded to the subscription destination.
  ForwardedLogEvents: The number of log events forwarded to the subscription destination.
• Check that there aren't errors when you stream log events to your destination.
  Review the following metrics:
  DeliveryErrors: The number of log events that CloudWatch Logs received an error for when forwarding data to the subscription destination.
  DeliveryThrottling: The number of log events that indicate CloudWatch Logs was throttled when data was forwarded to the subscription destination.
• If you have a dedicated Kinesis stream, then check the stream's metrics to confirm functionality.
• For issues with cross-account logging, see Troubleshooting your CloudWatch cross-account setup.