Problem Of Event Bridge pipes with KMS Encrypted SNS topic as Target

Question

Hello everyone,

We are using Event bridge pipes to integrate the DDB Streams and push events after enrichment to a SNS target

The KMS Key has following Key Policy

```
      KeyPolicy:
        Version: '2012-10-17'
        Statement:
          - Sid: Allow administration of the key
            Effect: Allow
            Principal:
              AWS:
                - !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action:
              - kms:Create*
              - kms:Describe*
              - kms:Enable*
              - kms:List*
              - kms:Put*
              - kms:Update*
              - kms:Revoke*
              - kms:Disable*
              - kms:Get*
              - kms:Delete*
              - kms:ScheduleKeyDeletion
              - kms:CancelKeyDeletion
              - kms:TagResource
              - kms:UntagResource
            Resource: '*'
          - Effect: Allow
            Principal:
              AWS:
                - !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action:
              - kms:Encrypt
              - kms:Decrypt
              - kms:ReEncrypt*
              - kms:GenerateDataKey*
              - kms:DescribeKey
            Resource: '*'
          - Effect: Allow
            Principal:
              Service:
                - pipes.amazonaws.com
            Action:
              - kms:Decrypt
              - kms:GenerateDataKey*
            Resource: "*"
```

As well we attached a Access Policy to SNS topic as bellow 
```
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Sid: Allow-SQS-Subscribe
            Principal:
              Service: sqs.amazonaws.com
            Action:
              - sns:Subscribe
            Resource: !Ref Topic
          - Effect: Allow
            Principal:
              Service: pipes.amazonaws.com
            Action:
              - sns:Publish
            Resource: "*"

```

The Pipe works well but we face Failures in Target publish 
The cloudwatch metrics shows us the errors on TargetStageFailed permanently 
The Pipe role has the following policy attached

```
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - "dynamodb:DescribeStream"
              - "dynamodb:GetRecords"
              - "dynamodb:GetShardIterator"
              - "dynamodb:ListStreams"
            Resource: '*'
          - Effect: Allow
            Action:
              - "sns:Publish"
            Resource: !Ref Topic
          - Effect: Allow
            Action:
              - lambda:InvokeFunction
            Resource:
              - !Ref LambdaArn
          - Effect: Allow
            Action:
              - sqs:ReceiveMessage
              - sqs:DeleteMessage
              - sqs:GetQueueAttributes
              - sqs:ChangeMessageVisibility
            Resource:
              - !Ref DeadLetterQueueArn
          - Effect: Allow
            Action:
              - sqs:SendMessage
            Resource:
              - !Ref DeadLetterQueueArn
          - Effect: Allow
            Action:
              - kms:Decrypt
              - kms:GenerateDataKey
            Resource: "*"
```
  
When Trying the deployed solution the enrichment is successfully passed but the EB rerun the enrichment and after investigation we found that with KMS the event bridge can not reach the Topic , if we remove the encryption everything goes well.

I believe that the problem is in EB side for any reason preventing the pipe reach out target , as i find no metric available in cloudwatch for the SNS Topic and in the list of topic in Cloudwatch it s not present.

There is no clear documentation and based on docs found on aws documentation everything is respected except we did not catch dome important part ;(

I appreciate any help and recommendation

**Update**

Some giving note for whom are struggling ,
* Take care and refine the policies if you use KMS Alias
* Check your IAM permission boundaries 
* Check CloudTrail to identify the problem 
* EventBridge will not mention you the root cause as with most of service integrations
* The KMS process takes place outside of EB and this is not an EB problem your can find
* Check as well your target service in cloudtrail

For me that was a problem of Permission boundaries enforced by Security team letting me just use the alias and not key directly

Accepted Answer

I just executed a quick test, and it worked with the following elements:

KMS Key policy: Make sure to have something like this in the policy:

```
{
    "Sid": "Allow access for EB Pipes",
    "Effect": "Allow",
    "Principal": {
        "Service": "pipes.amazonaws.com"
    },
    "Action": [
        "kms:GenerateDataKey*",
        "kms:Decrypt"
    ],
    "Resource": "*",
    "Condition": {
        "StringEquals": {
            "aws:SourceAccount": "111222333444"
        },
        "ArnLike": {
            "aws:SourceArn": "arn:aws:pipes:us-east-1:111222333444:pipe/test-pipe"
        }
    }
}
```

The service role for pipes should have the following permission:

```
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": [
				"kms:Decrypt",
				"kms:GenerateDataKey"
			],
			"Resource": "arn:aws:kms:us-east-1:111222333444:key/KEY-ID"
		}
	]
}
```

And last but not least, SNS Access Policy:

```
{
      "Sid": "allow_pipes",
      "Effect": "Allow",
      "Principal": {
        "Service": "pipes.amazonaws.com"
      },
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:us-east-1:111222333444:test-topic",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "111222333444"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:pipes:us-east-1:111222333444:pipe/test-pipe"
        }
      }
    }
```

Make sure to replace all account IDs and resource ARNs. 
In general, try to create policies as restrictive as possible (see "Conditions" blocks above).

I know that you already have many of those elements in your configuration, for the sake of completeness I still wanted to add them here. Hope this helps you resolving the issue!

PS: Maybe try an e-mail subscriber on the SNS topic, to make sure that the issue is not on the receiver side.

Answer

I don’t see any permissions in your KMS policy for the sqs queue?

Have you looked in cloudtrail to help debug the missing permissions and to confirm the service principle?

Problem Of Event Bridge pipes with KMS Encrypted SNS topic as Target

Relevant content