Building Intelligent Alerting Agent using Amazon Bedrock and STRANDS

Amazon Bedrock's AI capabilities can transform how you build intelligent observability systems to dynamically create and adjust the alarm configuration. This approach can significantly reduce alert fatigue while improving coverage of critical infrastructure gaps through AI-powered automation. In this post, we explain how to build an AI-powered observability system that automatically discovers monitoring blind spots, optimizes alert thresholds based on actual usage patterns, and generates infrastructure as code (IaC) for immediate deployment while maintaining human oversight.

Impact to your operations

You face several compounding challenges that make the case for AI-assisted alerting compelling: Alert fatigue is real and measurable. Static thresholds generate excessive false positives. When you receive hundreds of non-actionable alerts daily, you stop trusting the system. Critical alerts get lost in the noise, and event response times increase. If you've found yourself ignoring alerts because too many turned out to be false alarms, you understand this challenge.

Configuration drift is inevitable. Infrastructure changes constantly, new services launch, instances scale, traffic patterns shift. Monitoring configurations rarely keep pace. The result is a growing gap between what you monitor and what is recommended.

Manual threshold management doesn't scale. A single experienced site reliability engineer (SRE) might set excellent thresholds for a handful of services. But when you have hundreds of services across multiple accounts and AWS Regions, manual tuning becomes impractical. The "right" threshold for CPU utilization on a batch processing cluster differs significantly from a real-time API server, and both change over time.

Reactive by default. Most monitoring systems alert when a threshold is exceeded, meaning the problem has already occurred. The window for proactive intervention is missed. Inconsistent standards across teams. Without centralized enforcement, monitoring quality varies dramatically. Critical production databases may lack basic disk space alerts while development environments trigger alerts outside business hours.

Solution overview

This blog guides to implement AI-Driven Intelligent Alerting (AIDA), an architectural pattern that addresses each of these challenges through its four-agent architecture. Configuration drift is caught by continuous discovery scans. Manual threshold management is automated through AI-powered analysis of historical patterns. Reactive monitoring becomes proactive through gap analysis. And standards variation is eliminated through centralized baseline enforcement. AIDA automatically generates infrastructure as code (IaC) templates that transform AI insights into deployable monitoring configurations through your existing continuous integration and continuous delivery (CI/CD) pipelines. Most importantly, it supports compliance with AWS best practices while providing clear, natural language justifications for every recommendation. This approach scales monitoring expertise across your organization while maintaining essential human oversight for production reliability. Important: AIDA optimizes alert configurations based on historical data. It does not perform real-time anomaly detection on live metric streams. Before we dive into implementation, let's look at what the system produces. For example, when the Recommendation Agent processes a gap for an EC2 instance missing a CPUUtilization alarm, the output looks like this:

[json]
{
  "recommendation_id": "rec_001",
  "resource_id": "i-0abc123def456789",
  "resource_type": "EC2Instance",
  "gap_type": "missing_alarm",
  "metric_name": "CPUUtilization",
  "recommended_threshold": 85.0,
  "comparison_operator": "GreaterThanThreshold",
  "evaluation_periods": 2,
  "datapoints_to_alarm": 2,
  "period": 300,
  "statistic": "Average",
  "justification": "Based on 30-day analysis, this instance typically runs at 45-60% CPU during business hours with peaks up to 75%. The 85% threshold provides early warning while avoiding false positives during normal load spikes. Historical data shows only 2 occasions where CPU exceeded 85% - both during legitimate high-load events that required attention.",
  "confidence_score": 0.87,
  "historical_analysis": {
    "p50_cpu": 52.3,
    "p95_cpu": 74.1,
    "p99_cpu": 81.2,
    "max_cpu_30d": 89.4,
    "false_positive_risk": "low"
  },
  "iac_template": "cloudformation_alarm_template.yaml"
}

Architecture overview

The system uses Strands Agents, an open-source Python framework, to build four specialized agents, each responsible for a distinct phase of the observability optimization lifecycle. They operate in sequence, with human approval gates between analysis and implementation. We chose Strands Agents because it provides production-ready orchestration, built-in telemetry, and seamless integration with Amazon Bedrock's Converse API, reducing the boilerplate code you'd need to write from scratch.

Data flow

Trigger: A scheduled EventBridge rule or periodic manual trigger initiates the workflow. Discovery: The Discovery Agent scans CloudWatch alarms, metrics, and resource configurations across target accounts and Regions.

Gap Analysis: The Gap Analysis Agent compares discovered state against organizational baselines and AWS best practices.

Recommendation: The Recommendation Agent analyzes historical metric data, reasons about optimal thresholds, and generates prioritized recommendations with natural language justifications.

Human Approval: The system presents recommendations for review. No one deploys changes without explicit approval.

Implementation: You deploy approved changes as CloudFormation or Terraform templates through existing CI/CD pipelines.

Now that you understand how AIDA works conceptually, let's build it. The following sections guide you through implementing each agent, starting with the prerequisites.

AIDA Sample Code Access AWS Unified Operations and Incident Detection and Response customers can reach out to their Technical Account Managers to request for an AWS support version of AIDA sample code for intelligent observability deployments.

Prerequisites

Before you begin implementing this intelligent alerting system, verify you have the following: AWS Account setup: It needs an active AWS account with appropriate billing setup and can take estimated 2-4 hours for initial setup and testing, with estimated cost $5-10 for initial experimentation. Review AWS Pricing website for details. Centralized monitoring account setup using Amazon CloudWatch cross-account observability is recommended for production use.

Baseline configuration: Create a baseline monitoring configuration document in JSON format defining required alarms for different AWS services. This solution provides 5 baseline alarm configurations as per AWS best practice recommendations, with a few sample code examples for starters!

Following AWS Identity and Access Management (IAM) permissions are required: • Amazon CloudWatch: cloudwatch:DescribeAlarms, cloudwatch:GetMetricStatistics, cloudwatch:PutMetricAlarm • Amazon Bedrock: bedrock:InvokeModel, bedrock:InvokeModelWithResponseStream • Amazon EventBridge: events:PutRule, events:PutTargets • AWS CodePipeline: codepipeline:StartPipelineExecution (for IaC deployment)

Required tools:

[bash]
pip install strands-agents strands-agents-tools boto3

Implementation

Implement these agents as Python modules in your AWS environment. Each agent can run as an AWS Lambda function or as part of a containerized application. Follow these numbered steps to build your intelligent observability system:

Step 1: Set up the Discovery Agent The Discovery Agent builds a comprehensive inventory of your current monitoring state. We are using Claude Sonnet instead of other models because it provides a balance of reasoning capability and cost for infrastructure analysis tasks.

Step 2: Configure the Gap Analysis Agent The Gap Analysis Agent compares discovered state against organizational baselines stored in Parameter Store.

Step 3: Create the Recommendation Agent with threshold optimization This agent provides the most AI-powered value by analyzing historical metric data to suggest optimal thresholds.

Step 4: Implement human approval workflow Deploy a set of selected recommendations only with explicit human approval:

Step 5: Configure the Implementation Agent Deploy approved changes as IaC templates:

Enforce human approval technically by deploying the Implementation Agent in a separate AWS account with cross-account IAM roles that require explicit approval tokens.

Guardrails and safety considerations

An AI system that modifies production monitoring requires careful safety controls.

Human-in-the-loop is essential Deploy monitoring changes only with explicit human approval. The architecture enforces this through a mandatory approval gate that requires site reliability engineer (SRE) or operations team sign-off before alarm modifications reach production.

Threshold boundaries Even with AI optimization, enforce hard limits to prevent nonsensical recommendations:

[python]
THRESHOLD_BOUNDARIES = {
    "CPUUtilization": {"min": 50, "max": 95},
    "FreeStorageSpace": {"min": 1_073_741_824, "max": 107_374_182_400},  # 1GB to 100GB
    "Duration": {"min": 1000, "max": 300_000},  # Lambda: 1s to 300s
    "5XXError": {"min": 1, "max": 100},
}

Rollback capability and audit trail

Make every deployment reversible. The Implementation Agent should: • Tag newly created alarms with ManagedBy: AIDA and a version identifier • Store previous configurations before modification • Log every agent decision, including what data the agent analyzed, what threshold it recommended, who approved the change, and when it was deployed

Strands Agents provides built-in telemetry through OpenTelemetry that supports this requirement.

Clean up resources

To avoid ongoing charges, remove the following resources when you're finished experimenting: CloudWatch alarms: Delete the test alarms created by the system (tagged with ManagedBy: AIDA) EventBridge rules: Remove scheduled triggers and event rules IAM roles and policies: Remove service roles created for the agents

Next steps

Start with the Discovery Agent in a non-production environment to inventory current monitoring state. Join the community: Share your experiences and learn from others implementing AI-driven observability in the AWS DevOps Community and re:Post Observability channels. Explore advanced patterns: Once comfortable with the basics, explore cross-metric correlation and feedback loops through the AWS Observability Workshop. Scale gradually: Expand from single accounts to organization-wide monitoring standardization using AWS Organizations.

The most important architectural decision isn't which model to use or how to structure your agents—it's maintaining the human-in-the-loop approval process that validates AI recommendations before they affect production monitoring.

Conclusion

This post shows you how to implement intelligent observability and alerting using AI agents built with Amazon Bedrock and Strands Agents. This approach scales monitoring from a reactive operational burden into a proactive reliability advantage. Intelligent observability isn't about replacing human judgment with AI, it's about augmenting human expertise with AI's ability to process vast amounts of historical data, maintain consistency across hundreds of resources, and continuously re-evaluate configurations that humans set once and forget. The combination of Amazon Bedrock reasoning capabilities and the Strands Agents tooling framework provides a practical foundation for building these systems incrementally. Start with discovery, prove value, then expand.