Automating AWS DevOps Agent investigation from Incident Detection and Response alarms

Introduction

Note: This article is Part 2 of a series. In Part 1, you onboarded your workload to AWS Incident Detection and Response using the Incident Detection and Response CLI. This article connects the alarms to AWS DevOps Agent for automated investigation.

Organizations that use Incident Detection and Response benefit from a 5-minute response when critical production alarms activate. However, DevOps Agent provides an opportunity to go faster. While Incident Management Engineers (IMEs) investigate health on the AWS side and engage your on-call engineer, DevOps Agent performs autonomous investigation in parallel.

Route your Incident Detection and Response alarms to DevOps Agent through Amazon EventBridge to trigger autonomous investigation. Your on-call engineer can then join the bridge with metrics, dependencies, and correlated logs already in hand.

Solution overview

An EventBridge rule detects alarm state changes and invokes an AWS Lambda function that calls the DevOps Agent CreateBacklogTask API operation. This runs in parallel with Incident Detection and Response's existing 5-minute response without modifying your Incident Detection and Response configurations.

The solution consists of the following components, as seen in Figure 1:

• Amazon CloudWatch alarm or third-party APM event: Transitions to the ALARM state and sends an event to EventBridge.

• EventBridge rule: Matches the alarm pattern and invokes the Lambda function with the event payload.

• Lambda function: Calls the CreateBacklogTask operation on the DevOps Agent API, creating an INVESTIGATION task with HIGH priority.

• DevOps Agent: Begins the autonomous investigation and posts findings to the DevOps Agent console.

Prerequisites

To use this solution, you must meet the following requirements:

• An AWS Unified Operations or AWS Enterprise Support plan with an Incident Detection and Response entitlement

• Completed Incident Detection and Response onboarding with alarms deployed from Part 1

• An active AWS DevOps Agent Space in your AWS account

• AWS Command Line Interface (AWS CLI) version 2.34 or later

• AWS Identity and Access Management (IAM) permissions to create AWS CloudFormation stacks, Lambda functions, and EventBridge rules

Note: If you receive errors when you run AWS CLI commands, then see Troubleshooting errors for the AWS CLI.

Solution implementation

To implement this solution, you must first either create an DevOps Agent Space and associate observability tool with it. Or, confirm that you associated an observability with your existing Agent Space. Then, deploy a CloudFormation stack.

After you deploy your stack, test the flow end-to-end and monitor the integration.

Checking your DevOps Agent Space

If you already have an Agent Space, then run the following command to identify your DevOps Agent Space ID from the DevOps Agent console:

aws devops-agent list-agent-spaces --region your-region

Note: Replace your-region with your AWS Region.

Note the agentSpaceId value for the Agent Space that you want to use. You'll use the Agent Space ID when you deploy the CloudFormation stack.

Run the following list-associations command to check whether you associated an observability tool with your Agent Space:

aws devops-agent list-associations
--agent-space-id your-agent-space-id
--region your-region

Note: Replace your-agent-space-id with your Agent Space ID and your-region with your Region.

If you haven't created an Agent Space, then see Creating an Agent Space.

Make sure that you associate an AWS account with the Agent Space that has access to Amazon CloudWatch, AWS X-Ray, Amazon CloudWatch Logs, and AWS CloudTrail. These AWS services let DevOps Agent collect the signals that it needs during the investigation.

Note the Region where you deployed the Agent Space. In your CloudFormation deployment, the AgentSpaceRegion parameter refers to the Region where you deployed the Agent Space. However, Agent Spaces can investigate issues across Regions within the same account.

Note: Deploying this solution creates billable AWS resources, including Lambda functions and EventBridge rules. You incur charges for the duration that these resources exist. To avoid ongoing charges, follow the cleanup instructions after you test.

Deploying the CloudFormation stack

This implementation uses CloudFormation with inline Lambda code. The following sections break down the template into snippets for clarity. When you deploy your stack, combine the snippets into a single CloudFormation file.

Define the template parameters

In the following template, include your Agent Space ID, the Region where the Agent Space resides, and flexible alarm matching options:

AWSTemplateFormatVersion: '2010-09-09'
Description: >-
  Routes IDR alarms (CloudWatch and third-party APM) to AWS DevOps Agent
  for autonomous incident investigation

Parameters:
  AgentSpaceId:
    Type: String
    Description: Your DevOps Agent Space ID
  AgentSpaceRegion:
    Type: String
    Description: Region where the Agent Space resides
  AlarmNames:
    Type: CommaDelimitedList
    Default: ''
    Description: >-
      Comma-separated exact alarm names to route.
      Leave empty if using AlarmPrefix.
  AlarmPrefix:
    Type: String
    Default: ''
    Description: >-
      Prefix to match alarm names (e.g. "prod-" matches all
      alarms starting with that string). Leave empty if using AlarmNames.
  APMEventBusName:
    Type: String
    Default: ''
    Description: >-
      Custom EventBridge bus name deployed by the IDR CLI setup-apm command.
      Follows the naming convention {APMName}-AWSIncidentDetectionResponse-EventBus
      (e.g. "NewRelic-AWSIncidentDetectionResponse-EventBus").
      Leave empty if not using third-party APM.
  Boto3LayerArn:
    Type: String
    Description: ARN of Lambda Layer containing boto3 with DevOps Agent support

Conditions:
  HasAlarmList: !Not [!Equals [!Select [0, !Ref AlarmNames], '']]
  HasAlarmPrefix: !Not [!Equals [!Ref AlarmPrefix, '']]
  HasAPM: !Not [!Equals [!Ref APMEventBusName, ''

Create the IAM role

In the following template, the IAM role grants the Lambda function permission to write logs and call the DevOps Agent CreateBacklogTask API on your specific Agent Space:

Resources:
  TaskCreatorRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: CreateBacklogTask
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: aidevops:CreateBacklogTask
                Resource: !Sub >-
                  arn:aws:aidevops:${AgentSpaceRegion}:${AWS::AccountId}:agentspace/${AgentSpaceId}

Create the Lambda function

The Lambda function receives the alarm event from EventBridge and creates an investigation task. It handles both CloudWatch native alarms and third-party APM events, GenericAPMEvent.

Note: At time of publication, the Python 3.13 Lambda runtime includes boto3 1.40.x, which doesn't include the devops-agent service client. Run the following command to create a new Lambda layer.

mkdir -p python && pip install boto3 -t python/ && zip -r boto3-layer.zip python

aws lambda publish-layer-version --layer-name boto3-devops-agent

--zip-file fileb://boto3-layer.zip --compatible-runtimes python3.13

--query "LayerVersionArn" --output text

In the following example CloudFormation template, replace the Boto3LayerArn parameter with the ARN from the previous command’s output.

TaskCreatorFunction:
      Type: AWS::Lambda::Function
      Properties:
        FunctionName: idr-alarm-to-devopsagent-task
        Runtime: python3.13
        Handler: index.lambda_handler
        Role: !GetAtt TaskCreatorRole.Arn
        Timeout: 30
        ReservedConcurrentExecutions: 10
        Layers:
          - !Ref Boto3LayerArn
        Environment:
          Variables:
            AGENT_SPACE_ID: !Ref AgentSpaceId
            AGENT_SPACE_REGION: !Ref AgentSpaceRegion
        Code:
          ZipFile: |
            import json, os, boto3
            client = boto3.client(
                'devops-agent',
                region_name=os.environ['AGENT_SPACE_REGION']
            )
            def lambda_handler(event, context):
                print(f"Received event: {json.dumps(event)}")
                source = event.get("source", "")
                detail = event.get("detail", {})
                if source == "GenericAPMEvent":
                    # Third-party APM (Datadog, Dynatrace, Splunk)
                    incident_id = detail.get(
                        "incident-detection-response-identifier",
                        event.get("id")
                    )
                    title = f"APM Alert - {incident_id}"
                    description = detail.get(
                        "description", "Generic APM event received"
                    )
                else:
                    # CloudWatch native alarm
                    incident_id = detail.get("alarmName", event.get("id"))
                    title = f"CloudWatch Alarm: {incident_id}"
                    description = detail.get("state", {}).get(
                        "reason", "Alarm state changed"
                    )
                resp = client.create_backlog_task(
                    agentSpaceId=os.environ['AGENT_SPACE_ID'],
                    taskType='INVESTIGATION',
                    priority='HIGH',
                    title=title,
                    description=json.dumps({
                        "summary": description,
                        "incidentId": incident_id,
                        "incidentRegion": event.get("region"),
                        "event": event
                    })
                )
                print(f"Created task: {resp}")
                return {
                    'statusCode': 200,
                    'body': json.dumps({
                        'taskId': resp.get('taskId', resp.get('backlogTaskId', ''))
                    })
                }

Create the EventBridge rules

The CloudWatch template supports three alarm matching strategies. Use one or more of the following rules, depending on your setup.

CloudWatchAlarmRule: Use the following snippet to route only specific Incident Detection and Response onboarded alarms:

  CloudWatchAlarmRule:
    Type: AWS::Events::Rule
    Condition: HasAlarmList
    Properties:
      State: ENABLED
      EventPattern:
        source: [aws.cloudwatch]
        detail-type: ['CloudWatch Alarm State Change']
        detail:
          alarmName: !Ref AlarmNames
          state: { value: [ALARM] }
      Targets:
        - Arn: !GetAtt TaskCreatorFunction.Arn
          Id: DevOpsAgentTarget

  CloudWatchLambdaPermission:
    Type: AWS::Lambda::Permission
    Condition: HasAlarmList
    Properties:
      FunctionName: !Ref TaskCreatorFunction
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt CloudWatchAlarmRule.Arn

CloudWatchPrefixRule: Use the following snippet to route the alarms for a workload, such as alarms with a workload name prefix:

  CloudWatchPrefixRule:
    Type: AWS::Events::Rule
    Condition: HasAlarmPrefix
    Properties:
      State: ENABLED
      EventPattern:
        source: [aws.cloudwatch]
        detail-type: ['CloudWatch Alarm State Change']
        detail:
          alarmName: [{ prefix: !Ref AlarmPrefix }]
          state: { value: [ALARM] }
      Targets:
        - Arn: !GetAtt TaskCreatorFunction.Arn
          Id: DevOpsAgentTarget

  CloudWatchPrefixLambdaPermission:
    Type: AWS::Lambda::Permission
    Condition: HasAlarmPrefix
    Properties:
      FunctionName: !Ref TaskCreatorFunction
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt CloudWatchPrefixRule.Arn

GenericAPMRule: If you integrated a third-party APM tool with Incident Detection and Response using the Incident Detection and Response setup-apm CLI command, then use the following snippet. This rule adds a second consumer to the same Custom EventBus. The Incident Detection and Response managed rules route events to AWS Support. This rule routes the same events to DevOps Agent in parallel:

  GenericAPMRule:
    Type: AWS::Events::Rule
    Condition: HasAPM
    Properties:
      Name: !Sub 'da-apm-${APMEventBusName}'
      EventBusName: !Ref APMEventBusName
      EventPattern:
        source: [GenericAPMEvent]
        detail-type: ['ams.monitoring/generic-apm']
      State: ENABLED
      Targets:
        - Arn: !GetAtt TaskCreatorFunction.Arn
          Id: DevOpsAgentTarget

  APMLambdaPermission:
    Type: AWS::Lambda::Permission
    Condition: HasAPM
    Properties:
      FunctionName: !Ref TaskCreatorFunction
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceAccount: !Ref AWS::AccountId
      SourceArn: !GetAtt GenericAPMRule.Arn

Find your alarm names

Before you deploy the stack, identify the alarm names to match. How you identify the alarms depends on how you onboarded alarms in Part 1.

If you ran the awsidr create-alarms command, then use AlarmPrefix=IDR- in the deployment command.

If you ran the tag-based awsidr ingest-alarms command, then run the following command to rerun the same tag query:

aws resourcegroupstaggingapi get-resources 
  --tag-filters Key=Environment,Values=Production 
  --resource-type-filters cloudwatch:alarm 
  --query "ResourceTagMappingList[].ResourceARN" 
  --region your-region --output table

If you ran the awsidr ingest-alarms command for a manual ARN, then use the same alarm names that you provided during ingestion.

You can also reach out to your Technical Account Manager (TAM) to get the details of your onboarded alarms.

To deploy the setup with prefix matching, use the following snippet:

aws cloudformation deploy
  --template-file idr_devopsagent_event.yaml
  --stack-name idr-alarm-to-devopsagent
  --parameter-overrides
    AgentSpaceId=your-agent-space-id
    AgentSpaceRegion=your-agent-region
    AlarmPrefix="prod-"
  --capabilities CAPABILITY_IAM
  --region your-region

Note: Replace your-agent-space-id with your Agent Space ID, your-agent-region with the Region of your Agent Space, and your-region with your Region.

To deploy for both CloudWatch and third-party APM alarms, use the following snippet:

aws cloudformation deploy
  --template-file idr_devopsagent_event.yaml
  --stack-name idr-alarm-to-devopsagent
  --parameter-overrides
    AgentSpaceId=your-agent-space-id
    AgentSpaceRegion=your-agent-region
    AlarmPrefix="acme-payments-prod"
    APMEventBusName="{APMName}-AWSIncidentDetectionResponse-EventBus"
  --capabilities CAPABILITY_IAM
  --region your-region

Note: Replace your-agent-space-id with your Agent Space ID, your-agent-region with the Region of your Agent Space, and your-region with your Region. The APMEventBusName value must match the custom event bus that you deployed by the setup-apm Incident Detection and Response CLI command. To find the bus name, run awsidr ingest-alarms and select the APM option, or list event buses in your account that contain AWSIncidentDetectionResponse in the name.

Deploy the combined template

Combine all snippets into a single idr_devopsagent_event.yaml file to deploy the stack.

Testing the integration

After you deploy the stack, run the following set-alarm-state commands to test the end-to-end flow by triggering a synthetic alarm:

aws cloudwatch set-alarm-state
  --alarm-name "acme-payments-prod-ecs-cpu-high"
  --state-value ALARM
  --state-reason "Testing DevOps Agent event routing"
  --region your-region

Note: Replace your-agent-space-id with your Agent Space ID, your-agent-region with the Region of your Agent Space, and your-region with your Region.

Run the following command to verify that the Lambda function successfully invoked:

aws logs tail /aws/lambda/idr-alarm-to-devopsagent-task
  --since 5m
  --region your-region

Run the following command to confirm that the investigation task was created:

aws devops-agent list-backlog-tasks
  --agent-space-id your-agent-space-id
  --region your-region

Note: In the preceding commands, replace your-agent-space-id with your Agent Space ID and your-region with your Region.

The output contains a task with taskType set to INVESTIGATION, priority set to HIGH, and status showing IN_PROGRESS, that shows that DevOPs Agent is already investigating.

Adapt the solution

The Lambda function in this article creates investigation tasks from alarm events. However, you can adapt this solution for other use cases, such as the following ones:

• Custom priority mapping: Set CRITICAL priority for specific alarm names and HIGH for others.

• Enrichment: Add resource tags, account metadata, or on-call information to the task description before you create the task.

• Filtering: Add a short delay and recheck the alarm state to skip alarms that resolve within seconds, also known as flapping.

• Multiple Agent Spaces: Route different workloads to different Agent Spaces based on alarm name patterns.

Troubleshooting

If the integration doesn't work as you expect during your tests, then review the following common issues:

Symptom	Check	Fix
No investigation created	Lambda logs: aws logs tail /aws/lambda/idr-alarm-to-devopsagent-task --since 10m	Verify that the alarm name matches the prefix or list in stack parameters.
Lambda permission error	“AccessDeniedException” in logs	Verify that the IAM policy has aidevops:CreateBacklogTask on the correct Agent Space ARN
EventBridge not triggering	aws events describe-rule --name example-rule-name Note: Replace example-rule-name with the name of the rule.	Make sure that the rule state is ENABLED and the pattern matches the alarm name.
DevOps Agent not investigating	list-backlog-tasks shows PENDING_TRIAGE	Verify that you associated an account with the Agent Space and the IAM role is correct.

Cleanup

After you've finished testing, run the following delete-stack command to delete the CloudFormation stack:

aws cloudformation delete-stack \
  --stack-name idr-alarm-to-devopsagent \
  --region your-region

Note: Replace your-region with your Region.

The preceding command removes the Lambda function, EventBridge rules, and IAM role. Your Incident Detection and Response alarms and Agent Space aren't affected.

Note: Deleting the stack doesn't offboard your workload from Incident Detection and Response. The Incident Detection and Response monitoring and 5-minute response continues to operate independently.

Conclusion

By deploying a single CloudFormation stack, you connect your Incident Detection and Response onboarded alarms to DevOps Agent. Every alarm now triggers two parallel responses, a human-managed incident from Incident Detection and Response, a 5-minute engagement, and the DevOps Agent autonomous investigation. The same Lambda function handles both CloudWatch and third-party APM events.

This integration can reduce the time of first investigation findings to minutes. It provides your on-call engineer with immediate context before Incident Detection and Response engages and accelerates the incident bridge conversation with data-driven root cause signals.

To learn more about how AWS Support plans and offerings can help you get the most out of your AWS environment, see AWS Support. For implementation support and Incident Detection and Response configuration guidance, contact your TAM.

Related articles

• Implementing automated incident response with Incident Detection and Response and Dynatrace integration

• About AWS DevOps Agent

• IDR CLI on GitHub

About the author

Yomesh Shah Yomesh Shah is a Senior Solutions Architect at AWS. He brings over 25 years of experience helping enterprises maximize the value of their IT investments through optimization, automation, and process improvement. He currently helps AWS customers use scalable AWS Support solutions to enhance operational resilience and incident response capabilities. Yomesh holds a patent for the design of a Managed Services control plane in the cloud (US11856055B2).

Chris Parkyn Chris Parkyn has worked in the IT industry for over 30 years, spanning software development, infrastructure, and operations management. As a Senior Technical Account Manager at AWS, he partners with global financial services customers to strengthen operational resilience and optimize consumption.