How do I push CloudWatch logs across accounts to Amazon Data Firehose?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

To send CloudWatch logs to a Firehose stream in a different Region, the Region must support Firehose. 

Set up the destination account

In the resolution's commands, replace the following values with your values:

• 111111111111 with your destination account's ID
• us-east-1 with your Firehose Region
• us-west-2 with your Amazon Simple Storage Service (Amazon S3) bucket Region
• us-east-2 with your destination account's Region
• 222222222222 with your source account's ID
• -arn with the ARNs of the resources

To set up the destination account, complete the following steps:

1. Run the following command to create an Amazon S3 bucket:

   aws s3api create-bucket --bucket my-bucket --create-bucket-configuration LocationConstraint=us-west-2 --region us-west-2

   Note: Note the bucket's ARN from the output to use in a later step.

2. Create a trust policy that has the required permissions for Firehose to push data to Amazon S3:

   {
     "Statement": {
       "Effect": "Allow",
       "Principal": {
         "Service": "firehose.amazonaws.com"
       },
       "Action": "sts:AssumeRole",
       "Condition": {
         "StringEquals": {
           "sts:ExternalId": "111111111111"
         }
       }
     }
   }

3. Run the create-role command to create the AWS Identity and Access Management (IAM) role, and specify the trust policy:

   aws iam create-role \
       --role-name FirehosetoS3Role \
       --assume-role-policy-document file://~/TrustPolicyForFirehose.json

   Note: Note the role's ARN from the output to use in a later step.

4. To define the actions that Firehose can perform in the destination account, use the JSON editor to create a permissions policy similar to the following example:

   {
       "Version": "2012-10-17",
       "Statement": [
         {
           "Effect": "Allow",
           "Action": [
             "s3:AbortMultipartUpload",
             "s3:GetBucketLocation",
             "s3:GetObject",
             "s3:ListBucket",
             "s3:ListBucketMultipartUploads",
             "s3:PutObject"
           ],
           "Resource": [
             "arn:aws:s3:::my-bucket",
             "arn:aws:s3:::my-bucket/*"
           ]
         }
       ]
     }

5. Run the put-role-policy command to associate the permissions policy with the IAM role:

   aws iam put-role-policy --role-name FirehosetoS3Role --policy-name Permissions-Policy-For-Firehose --policy-document file://~/PermissionsForFirehose.json

6. Run the following command to create a destination delivery stream for Firehose:

   aws firehose create-delivery-stream --delivery-stream-name my-delivery-stream --s3-destination-configuration RoleARN='arn:aws:iam::111111111111:role/FirehosetoS3Role',BucketARN='arn:aws:s3:::my-bucket' --region us-east-1

   Note: Replace RoleARN and BucketARN with your role and bucket ARNs.
   When you deliver an Amazon S3 object to Firehose, a custom prefix is used in the timestamp namespace expression. You can specify an extra prefix at the beginning of the time format (yyyy/MM/dd/HH/). If the prefix ends with a forward slash (/), then it appears as a folder in the S3 bucket.

7. To check the DeliveryStreamDescription.DeliveryStreamStatus property, run the describe-delivery-stream command:

   aws firehose describe-delivery-stream --delivery-stream-name "my-delivery-stream" --region us-east-1

   To confirm that the stream is active, check the command's output.
   Example output:

   {
     "DeliveryStreamDescription": {
       "DeliveryStreamType": "DirectPut",
       "HasMoreDestinations": false,
       "DeliveryStreamEncryptionConfiguration": {
         "Status": "DISABLED"
       },
       "VersionId": "1",
       "CreateTimestamp": 1604484348.804,
       "DeliveryStreamARN": "arn:aws:firehose:us-east-1:111111111111:deliverystream/my-delivery-stream",
       "DeliveryStreamStatus": "ACTIVE",
       "DeliveryStreamName": "my-delivery-stream",
       "Destinations": [
         {
           "DestinationId": "destinationId-000000000001",
           "ExtendedS3DestinationDescription": {
             "RoleARN": "arn:aws:iam::111111111111:role/FirehosetoS3Role2test",
             "BufferingHints": {
               "IntervalInSeconds": 300,
               "SizeInMBs": 5
             },
             "EncryptionConfiguration": {
               "NoEncryptionConfig": "NoEncryption"
             },
             "CompressionFormat": "UNCOMPRESSED",
             "S3BackupMode": "Disabled",
             "CloudWatchLoggingOptions": {
               "Enabled": false
             },
             "BucketARN": "arn:aws:s3:::my-bucket"
           },
           "S3DestinationDescription": {
             "RoleARN": "arn:aws:iam::111111111111:role/FirehosetoS3Role2test",
             "BufferingHints": {
               "IntervalInSeconds": 300,
               "SizeInMBs": 5
             },
             "EncryptionConfiguration": {
               "NoEncryptionConfig": "NoEncryption"
             },
             "CompressionFormat": "UNCOMPRESSED",
             "CloudWatchLoggingOptions": {
               "Enabled": false
             },
             "BucketARN": "arn:aws:s3:::my-bucket"
           }
         }
       ]
     }
   }

   Note: Note the steam's ARN to use in a later step.

8. Create an additional trust policy to grant CloudWatch Logs the permission to put data into the Firehose stream. Add the Regions where the logs are pushed.
   Example policy:

   {
     "Version": "2012-10-17",
     "Statement": {
       "Effect": "Allow",
       "Principal": {
         "Service": "logs.us-east-2.amazonaws.com"
       },
       "Action": "sts:AssumeRole",
       "Condition": {
         "StringLike": {
           "aws:SourceArn": [
             "arn:aws:logs:us-east-2:sourceAccountId:*",
             "arn:aws:logs:us-east-2:recipientAccountId:*"
           ]
         }
       }
     }
   }

9. To create an additional IAM role to put data into the Firehose stream and specify the trust policy file, run the create-role command:

   aws iam create-role \
       --role-name CWLtoKinesisFirehoseRole \
       --assume-role-policy-document file://~/TrustPolicyForCWL.json

   Note: Note the role's ARN to use in a later step.

10. Create a permissions policy to define the actions that CloudWatch logs can perform in the destination account. Include the stream's ARN and the role's ARN.
    Example policy:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "firehose:ListDeliveryStreams",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "firehose:DescribeDeliveryStream",
                    "firehose:PutRecord",
                    "firehose:PutRecordBatch"
                ],
                "Resource": "arn:aws:firehose:us-east-1:111111111111:deliverystream/my-delivery-stream"
            }
        ]
    }

11. To associate the permissions policy with the role, run the put-role-policy command:

    aws iam put-role-policy --role-name CWLtoKinesisFirehoseRole --policy-name Permissions-Policy-For-CWL --policy-document file://~/PermissionsForCWL.json

12. To create a destination in the destination account for the source account to send logs to, run the put-destination command:

    aws logs put-destination --destination-name "myDestination" --target-arn "arn:aws:firehose:us-east-1:111111111111:deliverystream/my-delivery-stream" --role-arn "arn:aws:iam::111111111111:role/CWLtoKinesisFirehoseRole" --region us-east-2

    Note: You can create a destination for the delivery stream in any Region that supports Firehose. The Region where you create the destination must be same as the log source Region.

13. Create an access policy for the CloudWatch destination.
    Example policy:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "",
          "Effect": "Allow",
          "Principal": {
            "AWS": "222222222222"
          },
          "Action": "logs:PutSubscriptionFilter",
          "Resource": "arn:aws:logs:us-east-2:111111111111:destination:myDestination"
        }
      ]
    }

14. Run the following command to associate the access policy with the CloudWatch destination:

    aws logs put-destination-policy --destination-name "myDestination" --access-policy file://~/AccessPolicy.json --region us-east-2

15. To verify the destination, run the describe-destinations command:

    aws logs describe-destinations --region us-east-2

Set up the source account

Note: To set up the source account, you must be the account's IAM admin user or root user.

Complete the following steps:

1. Create a trust policy similar to the following example to grant Amazon Virtual Private Cloud (Amazon VPC) Flow Logs the permission to send data to the CloudWatch log group:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "",
         "Effect": "Allow",
         "Principal": {
           "Service": "vpc-flow-logs.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

2. Run the create-role command and specify the trust policy:

   aws iam create-role \
       --role-name PublishFlowLogs \
          --assume-role-policy-document file://~/TrustPolicyForVPCFlowLogs.json

   Note: Note the role's ARN from the output to use in a later step.

3. To define the actions that VPC Flow Logs can perform in the source account, create a permissions policy similar to the following example:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Action": [
           "logs:CreateLogGroup",
           "logs:CreateLogStream",
           "logs:PutLogEvents",
           "logs:DescribeLogGroups",
           "logs:DescribeLogStreams"
         ],
         "Effect": "Allow",
         "Resource": "*"
       }
     ]
   }

4. To associate the permissions policy with the IAM role, run the put-role-policy command:

   aws iam put-role-policy --role-name PublishFlowLogs --policy-name Permissions-Policy-For-VPCFlowLogs --policy-document file://~/PermissionsForVPCFlowLogs.json

5. To configure the destination for the flow logs, run the create-log-group command to create a CloudWatch log group:

   aws logs create-log-group --log-group-name vpc-flow-logs --region us-east-2

   Note: Replace us-east-2 with the Region where your Amazon VPC is located. 

6. To turn on VPC Flow Logs, run the create-flow-logs command:

   aws ec2 create-flow-logs --resource-type VPC --resource-ids vpc-12345678 --traffic-type ALL --log-group-name vpc-flow-logs --deliver-logs-permission-arn arn:aws:iam::222222222222:role/PublishFlowLogs --region us-east-2

   Note: Replace us-east-2 with the Region where your Amazon VPC is located. 

7. To subscribe the CloudWatch log group to Firehose in the destination account, run the put-subscription-filter command:

   aws logs put-subscription-filter --log-group-name "vpc-flow-logs" --filter-name "AllTraffic" --filter-pattern "" --destination-arn "arn:aws:logs:us-east-2:111111111111:destination:myDestination" --region us-east-2

   Note: Replace us-east-2 with the Region where your Amazon VPC is located. 
   To confirm that the logs are published, review the S3 bucket for new logs.

Related information

DeliveryStreamDescription