By using AWS re:Post, you agree to the AWS re:Post Terms of Use

How do I troubleshoot cross-account logging issues in CloudWatch?

3 minute read
0

I want to use my Amazon CloudWatch account to troubleshoot issues with cross-account logging.

Resolution

Incorrect target-level permissions can cause issues with cross-account logging.

Troubleshoot a Kinesis Data stream or Firehose stream as a destination target

To troubleshoot issues with streams to Amazon Kinesis Data Streams or Amazon Data Firehose, complete the following steps:

  1. Use the Kinesis console or the DescribeStream and DescribeDeliveryStream API calls to verify that your Kinesis data stream or Firehose stream is in an active state.

  2. Check that the CloudWatch log group and the Kinesis data stream are in the same AWS Region.

  3. Create an AWS Identity and Access Management (IAM) role with the correct trust permissions. The IAM role must allow CloudWatch Logs to send log events to a Kinesis data stream or Firehose stream.
    In the following example, the log data recipient account has an AWS account ID of 999999999999. The log data sender account ID is 111111111111. Make sure that account 111111111111 is allowed in the recipient account's trust policy for the Kinesis data stream or Firehose stream. 
    Example IAM trust policy for a Kinesis data stream or Firehose stream:

    {    "Statement": {
        "Effect": "Allow",
        "Principal": {
          "Service": "logs.region.amazonaws.com"
        },
        "Condition": {
          "StringLike": {
            "aws:SourceArn": [
              "arn:aws:logs:region:sourceAccountId:*",
              "arn:aws:logs:region:recipientAccountId:*"
            ]
          }
        },
        "Action": "sts:AssumeRole"
      }
    }

    Example IAM role policy for a Kinesis data stream:

    {    "Statement": [
        {
          "Effect": "Allow",
          "Action": "kinesis:PutRecord",
          "Resource": "arn:aws:kinesis:region:999999999999:stream/RecipientStream"
        }
      ]
    }
    
     IAM role policy for kinesis Firehose stream.
    
     {
      "Statement": [
        {      "Effect": "Allow",
          "Action": [
            "firehose:*"
          ],
          "Resource": [        "arn:aws:firehose:region:999999999999:*"
          ]
        }
      ]
    }
  4. Verify that the Regions and resource ARNs in your IAM policy are correct.

  5. When you configure the subscription filter for Kinesis Data Streams, don't select Firehose.

  6. After you start the stream, use the ForwardedBytes and ForwardedLogEvents CloudWatch metrics for the subscription filter. Confirm that the filter pattern is valid and matches the incoming log events.

  7. To confirm that there are no errors when you stream log events to the destination, check the DeliveryErrors and DeliveryThrottling metrics.
    Note: You might find a datapoint that confirms that the log event received an error when you stream log events to the destination.

  8. If you have a dedicated stream that provides insights for your Kinesis data stream functionality, then check the metrics for your Kinesis data stream. Or, check the metrics for your Firehose stream.

Troubleshoot an OpenSearch Service domain as a destination target

To troubleshoot issues with Amazon OpenSearch Service as a destination target, complete the following steps:

  1. Check that the OpenSearch Service domain allows either public access or Amazon Virtual Private Cloud (Amazon VPC) access.

  2. Make sure that the following IAM role policy has an AWS Lambda function that has an IAM execution role with a trust relationship for lambda.amazonaws.com:

    {
        "Version": "2012-10-17",
      "Statement": [
        {
          "Action": [
            "es:*"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:es:region:account-id:domain/target-domain-name/*"
        }
      ]
    }
  3. If the target OpenSearch Service domain uses VPC access, then check that the role has AWSLambdaVPCAccessExecutionRole policy attached.

Related information

Concepts

How do I create, configure, and troubleshoot a subscription filter to Kinesis using the CloudWatch console?

Streaming CloudWatch Logs data to Amazon OpenSearch Service

Create cross-account and cross-Region CloudWatch dashboards video

AWS OFFICIAL
AWS OFFICIALUpdated 4 months ago