How can I push CloudWatch logs across accounts to Amazon Data Firehose?

6 minute read

I want to stream Amazon CloudWatch logs from Amazon Data Firehose to another AWS account in a different AWS Region.

Resolution

To send CloudWatch logs to a Firehose stream in a different Region, the Region must support Firehose.

In the resolution's commands, replace the following values with your values:

111111111111 with your destination account's ID
us-east-1 with your Firehose Region
us-west-2 with your Amazon Simple Storage Service (Amazon S3) bucket Region
us-east-2 with your destination account's Region
222222222222 with your source account's ID
us-east2 with your CloudWatch log group Region
us-east-2 with your Amazon Virtual Private Cloud (Amazon VPC) Flow Logs Region
-arn with the ARNs of the resources

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Set up the destination account

Complete the following steps:

Create an Amazon S3 bucket:

aws s3api create-bucket --bucket my-bucket --create-bucket-configuration LocationConstraint=us-west-2 --region us-west-2

Note: Note the bucket's ARN from the output to use in a later step.

Create a trust policy that has the required permissions for Firehose to push data to Amazon S3:

{
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "Service": "firehose.amazonaws.com"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": {
        "sts:ExternalId": "111111111111"
      }
    }
  }
}

Run the create-role command to create the IAM role, and specify the trust policy:

aws iam create-role \
    --role-name FirehosetoS3Role \
    --assume-role-policy-document file://~/TrustPolicyForFirehose.json

Note: Note the role's ARN from the output to use in a later step.

To define the actions that Firehose can perform in the destination account, use the JSON editor to create a permissions policy:

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "s3:AbortMultipartUpload",
          "s3:GetBucketLocation",
          "s3:GetObject",
          "s3:ListBucket",
          "s3:ListBucketMultipartUploads",
          "s3:PutObject"
        ],
        "Resource": [
          "arn:aws:s3:::my-bucket",
          "arn:aws:s3:::my-bucket/*"
        ]
      }
    ]
  }

Run the put-role-policy command to associate the permissions policy with the IAM role:

aws iam put-role-policy --role-name FirehosetoS3Role --policy-name Permissions-Policy-For-Firehose --policy-document file://~/PermissionsForFirehose.json

Create a destination delivery stream for Firehose:

aws firehose create-delivery-stream --delivery-stream-name my-delivery-stream --s3-destination-configuration RoleARN='arn:aws:iam::111111111111:role/FirehosetoS3Role',BucketARN='arn:aws:s3:::my-bucket' --region us-east-1

Note: Replace RoleARN and BucketARN with your role and bucket ARNs.
When you deliver an S3 object to Firehose, a custom prefix is used in the timestamp namespace expression. You can specify an extra prefix at the beginning of the time format (yyyy/MM/dd/HH/). If the prefix ends with a forward slash (/), then it appears as a folder in the S3 bucket.

To check the DeliveryStreamDescription.DeliveryStreamStatus property, run the describe-delivery-stream command:

aws firehose describe-delivery-stream --delivery-stream-name "my-delivery-stream" --region us-east-1

To confirm that the stream is active, check the command's output:

{
  "DeliveryStreamDescription": {
    "DeliveryStreamType": "DirectPut",
    "HasMoreDestinations": false,
    "DeliveryStreamEncryptionConfiguration": {
      "Status": "DISABLED"
    },
    "VersionId": "1",
    "CreateTimestamp": 1604484348.804,
    "DeliveryStreamARN": "arn:aws:firehose:us-east-1:111111111111:deliverystream/my-delivery-stream",
    "DeliveryStreamStatus": "ACTIVE",
    "DeliveryStreamName": "my-delivery-stream",
    "Destinations": [
      {
        "DestinationId": "destinationId-000000000001",
        "ExtendedS3DestinationDescription": {
          "RoleARN": "arn:aws:iam::111111111111:role/FirehosetoS3Role2test",
          "BufferingHints": {
            "IntervalInSeconds": 300,
            "SizeInMBs": 5
          },
          "EncryptionConfiguration": {
            "NoEncryptionConfig": "NoEncryption"
          },
          "CompressionFormat": "UNCOMPRESSED",
          "S3BackupMode": "Disabled",
          "CloudWatchLoggingOptions": {
            "Enabled": false
          },
          "BucketARN": "arn:aws:s3:::my-bucket"
        },
        "S3DestinationDescription": {
          "RoleARN": "arn:aws:iam::111111111111:role/FirehosetoS3Role2test",
          "BufferingHints": {
            "IntervalInSeconds": 300,
            "SizeInMBs": 5
          },
          "EncryptionConfiguration": {
            "NoEncryptionConfig": "NoEncryption"
          },
          "CompressionFormat": "UNCOMPRESSED",
          "CloudWatchLoggingOptions": {
            "Enabled": false
          },
          "BucketARN": "arn:aws:s3:::my-bucket"
        }
      }
    ]
  }
}

Note: Note the steam's ARN to use in a later step.

Create an additional trust policy to grant CloudWatch Logs the permission to put data into the Firehose stream. Add the Regions where the logs are pushed:

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "Service": "logs.us-east-2.amazonaws.com"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringLike": {
        "aws:SourceArn": [
          "arn:aws:logs:us-east-2:sourceAccountId:*",
          "arn:aws:logs:us-east-2:recipientAccountId:*"
        ]
      }
    }
  }
}

To create an additional IAM role to put data into the Firehose stream and specify the trust policy file, run the create-role command:

aws iam create-role \
    --role-name CWLtoKinesisFirehoseRole \
    --assume-role-policy-document file://~/TrustPolicyForCWL.json

Note: Note the role's ARN to use in a later step.

Create a permissions policy to define the actions that CloudWatch Logs can perform in the destination account. Include the stream's ARN and the role's ARN:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "firehose:ListDeliveryStreams",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "firehose:DescribeDeliveryStream",
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "arn:aws:firehose:us-east-1:111111111111:deliverystream/my-delivery-stream"
        }
    ]
}

To associate the permissions policy with the role, run the put-role-policy command:

aws iam put-role-policy --role-name CWLtoKinesisFirehoseRole --policy-name Permissions-Policy-For-CWL --policy-document file://~/PermissionsForCWL.json

To create a destination in the destination account for the source account to send logs to, run the put-destination command:

aws logs put-destination --destination-name "myDestination" --target-arn "arn:aws:firehose:us-east-1:111111111111:deliverystream/my-delivery-stream" --role-arn "arn:aws:iam::111111111111:role/CWLtoKinesisFirehoseRole" --region us-east-2

Note: You can create a destination for the delivery stream in any Region where Firehose is supported. The Region where you create the destination must be same as the log source Region.

Create an access policy for the CloudWatch destination:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "222222222222"
      },
      "Action": "logs:PutSubscriptionFilter",
      "Resource": "arn:aws:logs:us-east-2:111111111111:destination:myDestination"
    }
  ]
}

Associate the access policy with the CloudWatch destination:

aws logs put-destination-policy --destination-name "myDestination" --access-policy file://~/AccessPolicy.json --region us-east-2

To verify the destination, run the describe-destinations command:

aws logs describe-destinations --region us-east-2

Set up the source account

Note: To set up the source account, you must be the account's IAM admin user or root user.

Complete the following steps:

Create a trust policy to grant Amazon VPC Flow Logs the permission to send data to the CloudWatch log group:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "vpc-flow-logs.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Run the create-role command and specify the trust policy:

aws iam create-role \
    --role-name PublishFlowLogs \
       --assume-role-policy-document file://~/TrustPolicyForVPCFlowLogs.json

Note: Note the role's ARN from the output to use in a later step.

To define the actions that VPC Flow Logs can perform in the source account, create a permissions policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

To associate the permissions policy with the IAM role, run the put-role-policy command:

aws iam put-role-policy --role-name PublishFlowLogs --policy-name Permissions-Policy-For-VPCFlowLogs --policy-document file://~/PermissionsForVPCFlowLogs.json

To configure the destination for the flow logs, run the create-log-group command to create a CloudWatch log group:

aws logs create-log-group --log-group-name vpc-flow-logs --region us-east-2

To turn on VPC Flow Logs, run the create-flow-logs command:

aws ec2 create-flow-logs --resource-type VPC --resource-ids vpc-12345678 --traffic-type ALL --log-group-name vpc-flow-logs --deliver-logs-permission-arn arn:aws:iam::222222222222:role/PublishFlowLogs --region us-east-2

To subscribe the CloudWatch log group to Firehose in the destination account, run the put-subscription-filter command:

aws logs put-subscription-filter --log-group-name "vpc-flow-logs" --filter-name "AllTraffic" --filter-pattern "" --destination-arn "arn:aws:logs:us-east-2:111111111111:destination:myDestination" --region us-east-2

To confirm that the logs are published, review the S3 bucket for new logs.

Related information

DeliveryStreamDescription

Topics

Analytics Internet of Things (IoT)

Relevant content

CloudWatch Logs sending to Firehose is missing data
hnguyen
asked 4 months ago
sending data from Cloudwatch 1 log stream of 20 to Firehose
rePost-User-6719888
asked 3 years ago
WAF logs to S3 and Cloudwatch?
atiu
asked 3 years ago
Kinesis data Firehose Data Transformation and Compression
Accepted Answer
Peter Eskandar
asked 2 years ago
Amazon Data Firehose Failing To Write To OpenSearch Serverless Collection
ayushi-189
asked 10 months ago
How do I turn on AWS WAF logging and send logs to CloudWatch, Amazon S3, or Firehose?
AWS OFFICIALUpdated 10 months ago
How do I set up cross-account streaming from Kinesis Data Firehose to Amazon OpenSearch Service?
AWS OFFICIALUpdated a year ago
How do I send my container logs to multiple destinations in Amazon ECS on AWS Fargate?
AWS OFFICIALUpdated a year ago
How do I troubleshoot cross-account logging issues in CloudWatch?
AWS OFFICIALUpdated 8 months ago
How can I send AWS WAF log to both CloudWatch logs and S3?
EXPERT
Lei Pei
published a year ago