AWS DevOps Agent: A Technical Deep Dive into Autonomous Incident Response

---

I've spent time investigating production incidents the hard way — correlating CloudWatch metrics, digging through deployment logs, and chasing down root causes at 2AM. AWS DevOps Agent changes that entirely. I'll walk you through the core components — Agent Spaces, the Web App, Skills, Incident Response, and Proactive Prevention — so you can get the most out of it in production.

---

What is AWS DevOps Agent?

AWS DevOps Agent is an automated incident response tool that reduces MTTR from hours to minutes by performing root cause analysis during incidents. It acts as an always-on DevOps expert by:

• Learning your environment — maps resources and their relationships
• Integrating with your tools — connects observability platforms, runbooks, code repos, and CI/CD pipelines
• Correlating data comprehensively — links telemetry, code, and deployment data across systems
• Understanding dependencies — maps relationships across multicloud and hybrid infrastructure

---

1. Agent Spaces

Agent Spaces are the foundational configuration unit — the scope boundary that defines what your DevOps Agent can see, access, and act on. Think of them as on-call boundaries translated into infrastructure access controls.

Key configuration elements:

• Details — configuration overview and scope definition (accounts, clusters, services)
• Capabilities — the integrated tools and observability platforms the agent can access
• Operator Access — permissions controlling who can interact with the agent and at what level
• EKS Access Entries — direct cluster connectivity for Kubernetes workloads
• Runbooks — operational playbooks the agent references during investigations
• Account Usage — consumption monitoring

Best practices:

• Think on-call boundaries when scoping Agent Spaces — start with one team or one service cluster
• Use Infrastructure as Code for repeatable, version-controlled deployments
• More integrations = more accurate root cause analysis; incomplete observability leads to incomplete investigations
• Agent Space scope is a two-way door — iterate as patterns emerge

---

2. DevOps Agent Topology

DevOps Agent sits between your alert sources and your infrastructure — correlating signals across every connected tool before surfacing a root cause. Understanding the topology helps you design Agent Spaces that reflect your actual operational boundaries.

Core layers:

2.1/ Control Plane — Investigation Engine The AI-powered analysis layer that receives alerts, orchestrates queries across capability providers, maps resource dependencies, and generates prescriptive findings. It examines deployment specs, not just metrics — this is what separates it from traditional monitoring.

2.2/ Agent Spaces — Logical Boundaries Each Agent Space defines the operational perimeter. Multiple Agent Spaces can run in parallel — scoped by team, service cluster, or environment (prod/dev/staging). Each maintains its own capability providers, IAM role, EKS access entries, and skills library independently.

2.3/ Capability Providers — Data Sources The tools the agent queries during investigations:

Observability — CloudWatch, Container Insights, Datadog, Dynatrace, Grafana, New Relic, Splunk Code & CI/CD — GitHub, GitLab, CodePipeline, CodeDeploy Incident management — PagerDuty, ServiceNow, Slack Infrastructure — EKS clusters, RDS, Lambda, EC2

Connectivity flow:

Alert fires
↓
Investigation Engine
↓
CloudWatch → EKS Cluster → Code Repo
↓
Root Cause Identified
↓
Remediation Steps → ServiceNow / Slack

Multi-account topology: In enterprise environments, the Agent Space lives in a central observability account and assumes cross-account IAM roles into each workload account — querying EKS clusters, CloudWatch logs, and Config history without replicating data or exposing credentials.

---

3. IAM Permissioning

DevOps Agent operates on a least-privilege, read-first model.

Create two IAM roles for DevOps Agent:

1/ **Agent Space Role** (`DevOpsAgentRole-AgentSpace`) — trusted by `aidevops.amazonaws.com`, attached `AIDevOpsAgentAccessPolicy` plus an inline policy for Resource Explorer service-linked role creation.   

2/ **Operator App Role** (`DevOpsAgentRole-WebappAdmin`) — trusted with `sts:AssumeRole` + `sts:TagSession`, attached `AIDevOpsOperatorAppAccessPolicy`, scoping access to investigations, recommendations, and chat via `AgentSpaceId` condition.

Agent Space - Role

Web App - Role

3.1/ Agent Space Execution Role (per Agent Space) Each Agent Space gets a dedicated IAM role scoped to its operational boundary.

Minimum read permissions:

cloudwatch:GetMetricData / cloudwatch:DescribeAlarms
logs:FilterLogEvents / logs:DescribeLogGroups
eks:DescribeCluster / eks:ListClusters
ec2:DescribeInstances
rds:DescribeDBInstances
config:GetResourceConfigHistory
cloudtrail:LookupEvents

For more information about the full list of actions, see the section called "IAM roles setup" and “DevOps Agent IAM permissions” in DevOps user guide.

3.2/ EKS Access Entries For Kubernetes workloads, DevOps Agent uses EKS Access Entries — IAM-based cluster access with no kubeconfig distribution or long-lived credentials.

Navigate to Agent Space → EKS Access Entries → select target cluster. The agent automatically creates an access entry with AmazonEKSViewPolicy (read-only), scoped to specific namespaces if needed. Fully auditable and revocable.

3.3/ Cross-Account Access For multi-account architectures, configure a trust relationship in each workload account:

{
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::CENTRAL_ACCOUNT:role/DevOpsAgentSpaceRole"
  },
  "Action": "sts:AssumeRole"
}

Security best practices:

a/ Separate Agent Spaces per environment (prod/dev/staging) — never share IAM roles across environments b/ Restrict EKS access to specific namespaces, not cluster-admin c/ Store third-party API keys (Datadog, PagerDuty) in AWS Secrets Manager — no static credentials d/ Enable CloudTrail logging for all agent API calls and set alarms for unexpected privilege escalation e/ Review cross-account trust relationships quarterly and remove unused capability providers

---

4. The Web App: DevOps Center

The DevOps Center is your operational command interface, surfacing three primary views:

1/ Incident Response → Investigations Root cause analyses with mitigation steps. Each investigation includes incident description, account/region context, timestamps, and prescriptive remediation guidance.

2/ Prevention Proactive recommendations based on historical incident patterns — surfacing recurring failures, misconfigurations, and technical debt before they become incidents.

3/ On-Demand SRE Natural language queries about your architecture — deployment history, active alarms, resource configurations, and incident patterns — without writing a single query.

---

5. Skills

Skills are how DevOps Agent learns your operational patterns over time.

Learned Skills — automatically built from investigation feedback. As your team reviews findings and marks them accurate or inaccurate, the agent refines its understanding of your environment's normal behavior.

Custom Skills — manually defined, targeted by agent type (incident, prevention, or SRE). Use these to encode organization-specific runbooks, escalation paths, or domain knowledge the agent wouldn't learn organically.

Navigate to Settings → Skills → Add Skill to create custom skills and attach them to specific Agent Spaces.

---

6. Incident Response: Autonomous Investigation

The moment an alert fires, the agent begins a systematic investigation — without waiting for a human to start the process.

Real-world EKS scenario — Product Catalog Service performance degradation:

The workshop used a Retail Store Sample Application with five microservices (UI, Catalog, Cart, Checkout, Orders). The Product Catalog Service degraded from 1–2 second load times to 8–10 seconds.

DevOps Agent investigation flow:

6.1/ Check pods in the catalog namespace and identify node associations
6.2/ Review CloudWatch Container Insights at the container level — not just pod level
6.3/ Analyze deployment specs, resource limits, and container configuration
6.4/ Root cause identified: Sidecar containers consuming per-container resource limits, throttling the main container

Key insight: Resource limits apply per-container, not per-pod. Pod-level metrics alone would have missed this entirely — container-level visibility is essential.

Key learnings from the scenario:.

• Sidecars can directly impact main container performance
• Always check for unexpected containers (sidecars, init containers, injected containers)
• DevOps Agent examines deployment specs, not just metrics — this is what separates it from traditional monitoring

---

7. Proactive Prevention

Beyond reactive response, DevOps Agent continuously analyzes historical patterns to shift your operations from reactive to predictive.

What prevention covers:

• Recurring failure patterns before they escalate to incidents
• Infrastructure misconfigurations and technical debt
• Prescriptive guidance on resource limits, dependency risks, and upgrade requirements
• Tracking implementation of recommendations over time

---

Best Practices for Production Deployment

Effective DevOps Agent deployment depends on how you configure your Agent Spaces. An Agent Space too narrow misses critical investigation context. One too broad introduces performance overhead and complexity. Here's how to get it right.

a/ Design Agent Space Architecture

Think on-call boundaries — scope Agent Spaces the same way you assign on-call responsibilities. Three proven patterns:

Single team, single application — one Agent Space per on-call group, separate production from non-production
Shared services / NOC teams — dedicated Agent Space scoped to shared infrastructure (databases, networking, centralized logging) used across multiple applications
Enterprise scale (100s of applications) — use Infrastructure as Code (AWS CDK or Terraform) to deploy Agent Spaces programmatically as part of application onboarding workflows

b/ Implement Your Agent Space

Use the AWS Console wizard for first-time setup or IaC templates for repeatable, multi-account deployments
Configure cross-account IAM roles in each workload account — the agent assumes these roles to query CloudWatch Logs, describe resources, and build application topology
Verify SCPs allow aidevops:* and bedrock:InvokeModel actions — a common failure point where setup completes but investigations fail silently

c/ Configure Integrations (Priority Order)

c.1/ CloudWatch — automatic via IAM, no additional config needed
c.2/ Observability tools — Datadog, Dynatrace, New Relic, Splunk for distributed tracing and APM
c.3/ Code repositories — GitHub/GitLab for deployment correlation and code context
c.4/ CI/CD pipelines — correlate incidents with deployment timing
c.5/ Communication channels — Slack and ServiceNow for real-time investigation updates and ticket management

For tools beyond built-in integrations, use webhooks (Grafana, Prometheus, PagerDuty) or custom MCP servers — noting MCP endpoints must be publicly accessible HTTPS, read-only tools only.

d/ Configure Access Controls

Scope IAM policies to specific Agent Space ARNs — not account-wide. Separate viewer, operator, and admin permissions:

{
  "Action": [
    "devopsagent:GetAgentSpace",
    "devopsagent:StartInvestigation",
    "devopsagent:GetInvestigation"
  ],
  "Resource": "arn:aws:devopsagent:us-east-1:123456789012:agentspace/EcommerceProd"
}

e/ Test and Iterate

Agent Space scope is a two-way door — start narrow, expand based on results:

Trigger a test investigation with a symptom description (e.g., "High latency on /api/checkout")
Observe which resources the agent queries
Add accounts if investigations lack context; add integrations if telemetry gaps exist; narrow scope if performance degrades

Key takeaways:

• Think on-call boundaries when scoping Agent Spaces
• Use IaC for consistent, repeatable deployments
• More integrations = more accurate root cause analysis
• Iterate — expand or narrow scope as investigation patterns emerge

---

Conclusion

AWS DevOps Agent moves incident response from manual firefighting to autonomous, predictive operations. Agent Spaces give you precise scope control. Skills let the agent learn your environment over time. The DevOps Center surfaces investigations, prevention recommendations, and on-demand SRE queries in one place. Start with one team, one service, measure MTTR improvement — and expand from there.

Resources

• AWS DevOps Agent User guide
• Announcing General Availability of AWS DevOps Agent
• Best Practices for Deploying AWS DevOps Agent in Production
• Accelerate incident resolution with AWS DevOps Agent- Now included with AWS Support plans

---