How do I create, configure, and troubleshoot a subscription filter to Kinesis using the CloudWatch console?

Short description

CloudWatch logs can be sent in near real-time to the same account or to cross-account Kinesis or Amazon Kinesis Data Firehose destinations. You can do this by using a subscription filter. The CloudWatch Logs console supports the destination and setup configuration.

For information on how to configure a subscription filter by using filter pattern syntax, see Filter and pattern syntax.

Resolution

Subscription configuration for a Kinesis data stream in the same or current account

Note: The CloudWatch log group Region and the Kinesis destination Region must be the same.

Before you create your subscription, do the following:

• Create a Kinesis stream, if you haven't already done so.
• Create an AWS Identify and Access Management (IAM) role that has the following trust policy and permissions:

To create a custom IAM role and role policy, do the following:

1.    Open the IAM console with your user that has administrator permissions.

2.    In the navigation pane, choose Policies.

3.    In the content pane, choose Create policy.

4.    Copy and paste the following JSON policy document into the JSON tab.

Role permission

{
  "Statement": [{
    "Effect": "Allow",
    "Action": "kinesis:PutRecord",
    "Resource": "arn:aws:kinesis:<REGION>:<ACCOUNT_ID>:stream/<STREAM_NAME>"
  }]
}

5.    Open the IAM console.

6.    In the navigation pane of the console, choose Roles and then choose Create role.

7.    Choose the Custom trust policy role type.

8.    In the Custom trust policy section, enter or paste the custom trust policy for the role. For more information, see Creating IAM policies.

9.    Choose Next.

10.    Select the custom IAM policy created in the previous step.

11.    Choose Next, Create a Role

Trust policy

{
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "Service": "logs.region.amazonaws.com"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringLike": {
        "aws:SourceArn": "arn:aws:logs:<REGION>:<ACCOUNT_ID>:*"
      }
    }
  }
}

After you create a Kinesis stream and an IAM role, you can create a subscription filter:

1.    Open the CloudWatch console.

2.    Select Log Group.

3.    Select to Action, Subscription Filters.

4.    To configure the Destination choose Create Kinesis Subscription Filter.

5.    Select Current Account.

6.    Select your Kinesis data stream from the dropdown list.

7.    Select the IAM role that you previously created.

8.    Review the Distribution method:

By Log Stream - Verifies that downstream consumers can aggregate log events by log stream, but might be less efficient. This method might also incur higher streaming costs because it requires more shards.

Random - Distributes the load across Kinesis stream shards, but downstream consumers can't aggregate log events by log stream.

9.    Configure Log format and filters:

Select the log format. The format can be Amazon VPC Flow Logs, AWS CloudTrail, or AWS Lambda for logs published by Amazon VPC, CloudTrail or Lambda. Or, you can choose JSON, Space delimited, or Other, depending on your incoming log events.

Define the filter pattern in the Subscription filter pattern section.

Enter a name of your subscription filter.

10.    Verify your pattern with the existing log event data

11.    After verifying, select Start Streaming.

12.    (Optional) Verify that your data stream works by validating the flow of log events.

Configuration for a cross-account Kinesis data stream destination

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

You can deliver CloudWatch Logs events to Kinesis data streams in different AWS accounts and Regions. To do this, set up cross-account log data sharing with subscriptions while specifying the AWS Region, as shown in the following example.

Note: In this example, CloudWatch Logs in the us-east-1 Region are delivered to another AWS user's Kinesis data stream in us-west-2 . The log data recipient's AWS account ID is 222222222222, and the log data sender AWS account ID is 111111111111.

Create a destination data stream in the recipient account 222222222222

1.    Create a destination data stream in Kinesis in the data recipient account with an IAM role and trust policy.

2.    Use the create-stream command to create the data stream. Make sure to specify the --region. For example, this command creates the data stream YourStreamName in us-west-2:

aws kinesis create-stream --stream-name "YourStreamName" --shard-count 1 --region us-west-2

3.    Use the describe-stream command to check the StreamDescription.StreamStatus property. Make sure to specify the --region. For example, this command checks the stream YourStreamName in us-west-2:

aws kinesis describe-stream --stream-name "YourStreamName" --region us-west-2

4.    Use the put-destination command to create the CloudWatch Logs destination. Set the --region for the --role-arn to the same Region as the source CloudWatch logs. For example, the following command must run in the recipient/destination account 222222222222 that creates the log destination in the recipient account 222222222222 in us-east-1:

aws logs put-destination \
  --destination-name "testDestination" \
  --target-arn "arn:aws:kinesis:us-west-2:222222222222:stream/YourStreamName" \
  --role-arn "arn:aws:iam::222222222222:role/YourIAMRoleName" --region us-east-1

Create a subscription filter in the source account 111111111111

To create the subscription filter, do the following:

1.    Select the log group.

2.    Select Action, Subscription Filters.

3.    To select the destination, choose from the following options:

Create Kinesis Subscription Filter: Creates Subscription filter for a Kinesis data stream Destination

4.    Select Another Account, because the destination is cross-account.

5.    For a cross-account Kinesis or Kinesis Firehose destination, provide the Destination ARN.

6.    Select the Distribution method:

By Log Stream - Verifies that downstream consumers can aggregate log events by log stream, but might be less efficient. This method might also incur higher streaming costs because it requires more shards.

Random - Distributes the load across Kinesis stream shards, but downstream consumers can't aggregate log events by log stream.

7.    Configure Log format and filters:

Select the log format. The format can be Amazon VPC Flow Logs, CloudTrail, or AWS Lambda for logs published by Amazon VPC, CloudTrail, or Lambda. Or, you can choose JSON, Space delimited, or Other, depending on your incoming log events.

Define the filter pattern in the Subscription filter pattern section.

Enter a name of your subscription filter.

8.    Verify your pattern with the existing log event data

9.    After verifying, select Start Streaming.

10.    (Optional) Verify that your data stream works by validating the flow of log events.

Troubleshooting

• Make sure that your Kinesis stream is in the Active state and that you can view it on Kinesis console. Or, you can use the DescribeStream API call.
• Verify that the CloudWatch log group and Kinesis data stream Region are the same.
• Make sure that there is an IAM role that has trust permissions for logs.yourregion.amazonaws.com and allows the permission kinesis:putrecords.
• Verify that the Regions and resources in the IAM policy are correct.
• Make sure that you didn't select Kinesis Firehose when configuring a subscription filter for Kinesis data stream.
• After you start streaming, verify that the metrics for the subscription filter confirm that the filter pattern is valid and matches the incoming log events. To do this, review the following metrics: ForwardedBytes: The volume of log events in compressed bytes forwarded to the subscription destination. ForwardedLogEvents: The number of log events forwarded to the subscription destination.
• To verify that there aren't errors when streaming log events to your destination, verify the following metrics: DeliveryErrors: The number of log events that CloudWatch Logs received an error for when forwarding data to the subscription destination. DeliveryThrottling: The number of log events that indicate CloudWatch Logs was throttled when forwarding data to the subscription destination.
• If you have a dedicated Kinesis stream, check the stream's metrics to confirm functionality.
• To troubleshoot cross-account logging, see Troubleshooting your CloudWatch cross-account setup.

---