EventBridge DeadLetterConfig Drift False Positive

0

I have the following CloudFormation template that notifies users of AWS Health events via an SNS topic:

AWSTemplateFormatVersion: "2010-09-09"
Description: Subscribe to account-specific and global events from AWS Health
Parameters:
  Email:
    Type: String
    Description: The email address that will receive SNS notifications for the health alerts
    # Simple regex from https://stackoverflow.com/a/201378
    AllowedPattern: "(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|\"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*\")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])"
Resources:
  EventBridgeRule:
    Type: AWS::Events::Rule
    Properties:
      Name: USEast1HealthRule
      Description: Push AWS Health notifications to an SNS topic
      EventPattern:
        source:
          - aws.health
        detail-type:
          - AWS Health Event
        detail:
          service:
            # Services to get notified for health alerts
            - CLOUDFORMATION
            - S3
            - CLOUDFRONT
            - APIGATEWAY
            - LAMBDA
            - DYNAMODB
            - IAM
            - CODEBUILD
            - CODEDEPLOY
            - CODEPIPELINE
            - KMS
            - SQS
            - ECR
            - COGNITO
            - CLOUDWATCH
            - XRAY
      State: ENABLED
      Targets:
        - Id: EmailTarget
          Arn: !Ref SNSTopic
          # Failed events are retried up to 185 times within 24 hours by default
          DeadLetterConfig:
            Arn: !GetAtt DLQ.Arn
  SNSTopic:
    Type: AWS::SNS::Topic
    Properties:
      # Encrypt using the default SNS SSE key
      # Key aliases: aws kms list-aliases
      KmsMasterKeyId: alias/aws/sns
      Subscription:
        - Endpoint: !Ref Email
          Protocol: email
      TopicName: AWSHealthNotifications
  SNSTopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties:
      PolicyDocument:
        Statement:
          # Allow EventBridge to send email alerts using SNS
          - Sid: AllowEventBridgeServicePrincipalWriteOnly
            Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: sns:Publish
            Resource: !Ref SNSTopic
            Condition:
              ArnEquals:
                "aws:SourceArn": !GetAtt EventBridgeRule.Arn
          - Sid: AllowSSLRequestsOnly
            Effect: Deny
            Principal: "*"
            Action: sns:Publish
            Resource: !Ref SNSTopic
            Condition:
              Bool:
                "aws:SecureTransport": false
      Topics:
        - !Ref SNSTopic
  DLQ:
    Type: AWS::SQS::Queue
    DeletionPolicy: Delete
    UpdateReplacePolicy: Delete
    Properties:
      # Defaults:
      # DelaySeconds (delay queue) = 0s (time before messages appear in the queue)
      # MessageRetentionPeriod = 4 days (345,600s) (time messages stays in the queue)
      # VisibilityTimeout = 30s (time before messages reappear in the queue)
      MessageRetentionPeriod: 345600
      ReceiveMessageWaitTimeSeconds: 5 # long polling if > 0
      SqsManagedSseEnabled: true
  DLQPolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      # Allow EventBridge to send messages to the DLQ
      PolicyDocument:
        Statement:
          - Sid: AllowEventBridgeServicePrincipalWriteOnly
            Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: sqs:SendMessage
            Resource: !GetAtt DLQ.Arn
            Condition:
              ArnEquals:
                "aws:SourceArn": !GetAtt EventBridgeRule.Arn
          - Sid: AllowSSLRequestsOnly
            Effect: Deny
            Principal: "*"
            Action: "sqs:*"
            Resource: !GetAtt DLQ.Arn
            Condition:
              Bool:
                "aws:SecureTransport": false
      Queues:
        - !Ref DLQ
Outputs:
  DLQURL:
    Description: DLQ URL for EventBridge
    Value: !Ref DLQ

When I detect drift on this stack, I get the following output:

{
    "StackResourceDrifts": [
        {
            "StackId": "arn:aws:cloudformation:us-east-1:***:stack/Health-Alert-Stack/f035cfc0-162a-11ee-a14e-12f8f4f0e2d3",
            "LogicalResourceId": "EventBridgeRule",
            "PhysicalResourceId": "USEast1HealthRule",
            "ResourceType": "AWS::Events::Rule",
            "ExpectedProperties": "{\"Description\":\"Push AWS Health notifications to an SNS topic\",\"EventPattern\":\"{\\\"detail-type\\\":[\\\"AWS Health Event\\\"],\\\"source\\\":[\\\"aws.health\\\"],\\\"detail\\\":{\\\"service\\\":[\\\"CLOUDFORMATION\\\",\\\"S3\\\",\\\"CLOUDFRONT\\\",\\\"APIGATEWAY\\\",\\\"LAMBDA\\\",\\\"DYNAMODB\\\",\\\"IAM\\\",\\\"CODEBUILD\\\",\\\"CODEDEPLOY\\\",\\\"CODEPIPELINE\\\",\\\"KMS\\\",\\\"SQS\\\",\\\"ECR\\\",\\\"COGNITO\\\",\\\"CLOUDWATCH\\\",\\\"XRAY\\\"]}}\",\"Name\":\"USEast1HealthRule\",\"State\":\"ENABLED\",\"Targets\":[{\"Arn\":\"arn:aws:sns:us-east-1:***:AWSHealthNotifications\",\"DeadLetterConfig\":{\"Arn\":\"arn:aws:sqs:us-east-1:***:Health-Alert-Stack-DLQ-8Q2i6BSM97kI\"},\"Id\":\"EmailTarget\"}]}",
            "ActualProperties": "{\"Description\":\"Push AWS Health notifications to an SNS topic\",\"EventPattern\":\"{\\\"detail-type\\\":[\\\"AWS Health Event\\\"],\\\"source\\\":[\\\"aws.health\\\"],\\\"detail\\\":{\\\"service\\\":[\\\"CLOUDFORMATION\\\",\\\"S3\\\",\\\"CLOUDFRONT\\\",\\\"APIGATEWAY\\\",\\\"LAMBDA\\\",\\\"DYNAMODB\\\",\\\"IAM\\\",\\\"CODEBUILD\\\",\\\"CODEDEPLOY\\\",\\\"CODEPIPELINE\\\",\\\"KMS\\\",\\\"SQS\\\",\\\"ECR\\\",\\\"COGNITO\\\",\\\"CLOUDWATCH\\\",\\\"XRAY\\\"]}}\",\"Name\":\"USEast1HealthRule\",\"State\":\"ENABLED\",\"Targets\":[{\"Arn\":\"arn:aws:sns:us-east-1:***:AWSHealthNotifications\",\"Id\":\"EmailTarget\"}]}",
            "PropertyDifferences": [
                {
                    "PropertyPath": "/Targets/0/DeadLetterConfig",
                    "ExpectedValue": "{\"Arn\":\"arn:aws:sqs:us-east-1:***:Health-Alert-Stack-DLQ-8Q2i6BSM97kI\"}",
                    "ActualValue": "null",
                    "DifferenceType": "REMOVE"
                }
            ],
            "StackResourceDriftStatus": "MODIFIED",
            "Timestamp": "2023-08-01T02:24:13.184000+00:00"
        }
    ]
}

CloudFormation is detecting drift on the DeadLetterConfig of the EventBridge target, but I see in the management console and the CLI that the DLQ is configured:

aws events list-targets-by-rule --rule USEast1HealthRule:

{
    "Targets": [
        {
            "Id": "EmailTarget",
            "Arn": "arn:aws:sns:us-east-1:***:AWSHealthNotifications",
            "DeadLetterConfig": {
                "Arn": "arn:aws:sqs:us-east-1:***:Health-Alert-Stack-DLQ-8Q2i6BSM97kI"
            }
        }
    ]
}

I tried the following to no avail, making me think this is a false positive from CloudFormation:

  • I waited a few days to see if the drift error would go away, but it didn't.
  • I'm running the drift analysis as an admin user, so there shouldn't be any access issues.
  • I double-checked that everything was deployed in the us-east-1 region.
  • I tried commenting and uncommenting the DeadLetterConfig block to "delete" the resource, but the drift error remained.
  • I deployed the same stack to a sandbox account and created a similar stack for Trusted Advisor events, and I'm getting the same drift error.
1 Answer
0
Accepted Answer

Hi, yes, it seems to be an issue with improper drift detection. The best is to open a support case via the AWS console: Service Teams usually do not monitor re:Post for such issue. You will get proper follow-up on your problem via the Support case.

In the meantime, you may want to apply the CFN protection for "false drift detection": see https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/resource-type-model-false-drift.html

You can use this mechanism to at least hide the current drift issue until it gets fixed.

Best, Didier

profile pictureAWS
EXPERT
answered 9 months ago
  • Ok, thank you. I thought to post here since the docs mentioned how to report false positives:

    If you encounter any false positive, send us your comments using the feedback link in the CloudFormation console, or reach out to us through the AWS forums at https://forums.aws.amazon.com/.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions