Skip to content

Alarming at scale: Best practices for centralized AWS observability

10 minute read
Content level: Advanced
7

This article provides prescriptive guidance for customers reaching their Amazon CloudWatch Metrics Insights 200 alarm quota per AWS Region. It explains the challenge and recommends architectural approaches to distribute alarm capacity and implement scalable alarm design patterns, including using CloudWatch Observability Access Manager (OAM) for cross-account monitoring, Metrics Insights query consolidation, composite alarm hierarchies, and environment-based account segmentation.

Introduction

As organizations scale on AWS, the number of accounts, services, and resources grow exponentially. This creates a common challenge that's often raised during operational reviews. Service quotas for each account become a bottleneck for comprehensive monitoring, particularly the 200 Metrics Insights query-based alarms quota for each Region.

AWS Support and Technical Account Managers (TAMs) regularly help customers navigate this challenge. The patterns in this article are drawn from real-world engagements where Support teams helped customers redesign their monitoring architecture to scale sustainably.

When customers reach their alarm quotas and resources go unmonitored, the consequences are tangible. Security events slip through undetected, compliance gaps surface during audits, and cost anomalies escalate into significant budget overruns. For organizations that operate at scale, a well-architected observability strategy is essential to avoid these risks and control monitoring costs.

Customers with hundreds of microservices, dynamic auto-scaling environments, and multiple application teams that share production accounts frequently reach their alarm quotas. Individual alarm optimizations, such as query consolidation and composite alarms, can provide relief. However, the most effective long-term solution is to use a dedicated account to monitor across accounts.

Problem statement

Customers that operate at scale encounter the following constraints:

  • Each account has a quota of 200 Metrics Insights alarms for each Region that you can't adjust.
  • Organizations with hundreds of microservices, multi-AZ deployments, auto-scaling environments, and multiple teams that share a single production account quickly reach this quota.

Prerequisites

Required configurations

Before you implement the cross-account monitoring architecture in this article, configure the following requirements:

Optional configurations

  • Source accounts configured to share metrics, logs, or traces based on the monitoring needs of each account.

  • Administrative access to the designated monitoring accounts to create alarms, dashboards, and OAM sinks.

Solution

The solution uses CloudWatch OAM to establish a centralized monitoring account that receives metrics, logs, and traces from multiple source accounts. Create Metrics Insights alarms in the monitoring account to multiply your available alarm capacity and gain unified operational visibility.

The architecture consists of the following three tiers:

  • Monitoring Accounts: A central account that hosts cross-account alarms, dashboards, and investigations.
  • OAM Sink: An attachment point in the monitoring account where source accounts connect to share observability data. To create a sing, see CreateSink.
  • Source Accounts: Workload accounts that share metrics, logs, and traces with the monitoring account. To create a link, see CreateLink.

Note: There are no extra charges for shared logs and metrics, and the first trace copy is free. For more information about pricing, see Amazon CloudWatch pricing. For information about cross-account observability setup and costs, see Monitor across accounts and Regions.

Recommendations

Recommendation 1: Consolidate alarms with Metrics Insights queries

Instead of creating one alarm for each resource, use one Metrics Insights alarm to monitor an entire fleet. A Metrics Insights query can process up to 10,000 metrics in a single alarm evaluation.

For example, instead of 50 individual CPU alarms for 50 Amazon Elastic Compute Cloud (Amazon EC2) instances, create one Metrics Insights alarm:

SELECT MAX(CPUUtilization) FROM SCHEMA("AWS/EC2", InstanceId) GROUP BY InstanceId ORDER BY MAX() DESC LIMIT 50

For instructions, see Creating a Metrics Insights CloudWatch alarm. You can also create an alarm based on a Multi Time Series Metrics Insights query to monitor fleet-wide conditions.

Result: Reduces alarm count from N (number of resources) to 1 for each fleet. An Metric Insights alarm doesn't require manual updates because it dynamically adapts as resources come and go. For more information, see Optimizing alarm lifecycle with Amazon CloudWatch Metrics Insights alarms.

Recommendation 2: Use composite alarms for aggregation

Combine multiple alarms into one composite alarm to aggregate monitoring over an entire group of resources. A composite alarm triggers only when multiple conditions are simultaneously met, so you receive fewer notifications without reducing monitoring coverage.

For example, a "Service Health Critical" composite alarm triggers only when all of the following conditions are true:

  • Error rate that's greater than 5%
  • Latency P99 that's greater than 2 seconds
  • Healthy host count that's less than 2

Key advantage: Composite alarms don't have the 200 Metrics Insights quota. They're ideal for aggregation without using Metrics Insights alarm capacity.

To create a composite alarm, see Create a composite alarm. For more information about this approach, see Improve monitoring efficiency using Amazon CloudWatch composite alarms.

Recommendation 3: Use CloudWatch centralization rules for cross-account telemetry

CloudWatch centralization rules automatically replicate metrics and logs from multiple member accounts into a single destination account within your organization. You define rules that specify which accounts, Regions, and log groups to centralize, and the service automatically handles replication.

Unlike OAM, which provides query-level access, centralization rules give you full data ownership in the destination account. Centralization rules are good for customers who require centralized alarming and analysis.

Key advantages for alarm-centric use cases:

  • Data ownership: The destination account owns the centralized data, enabling full CloudWatch feature support for Metrics Insights alarms, anomaly detection, metric streams, and PromQL queries.
  • No query federation overhead: Data is physically replicated, so all alarms and dashboards operate on local data with no cross-account API operations at query time.
  • Source metadata preservation: Centralized metrics include :@aws.account and :@aws.region dimensions for Metrics Insights, or @aws.account and @aws.region attributes for PromQL so that you can filter by source.
  • Built-in resiliency: The optional backup Region configuration keeps log and metric data available during Regional outages.
  • Flexible scoping: Rules can target the entire organization, specific organizational units (Ous), or individual accounts, with the ability to filter log groups using LIKE/NOT LIKE patterns.

For more information, see Cross-account cross-Region metrics centralization.

When to use centralization rules instead of OAM:

  • Centralization rules: Use centralization rules when you require Metrics Insights alarms on cross-account data, full feature support for metric math and anomaly detection, or data residency in a specific account for compliance.
  • OAM: Use OAM when you require cross-account trace visibility, want to query data in place without replication, or require real-time federation across accounts.

Recommendation 4: Create separate monitoring accounts for each environment

Rather than keeping all alarms in one monitoring account, instead create dedicated monitoring accounts that you segment by environment. Each monitoring account receives its own 200 Metrics Insights alarms quota for each Region, which multiplies your capacity.

To set up cross-account monitoring, see Link monitoring accounts with source accounts. To set up a complete implementation pattern, see Centralize monitoring by using Amazon CloudWatch Observability Access Manager.

Recommended segmentation

Monitoring accountSource accountsAlarm focus
Production monitoringAll production accountsBusiness-critical service-level objectives (SLOs), revenue-impacting alerts, uptime monitoring
Non-production monitoringDevelopment, QA, staging accountsPerformance regression detection, development workflow alerts
Security monitoringAll accountsSecurity and compliance alarms, Amazon GuardDuty and AWS Config aggregation

Environment-based segmentation provides the following benefits:

  • Quota multiplication: With three monitoring accounts, you have 3 × 200 = 600 Metrics Insights alarms available for each Region.
  • Blast radius isolation: A misconfigured alarm that's in production monitoring can't affect development workflows.
  • Access control: The on-call team accesses production monitoring and developers access non-production monitoring.
  • Cost attribution: CloudWatch monitoring costs are separately tracked for each environment tier.
  • Differentiated thresholds: Production alarms are at 80 percent CPU and development at 90 percent, with no collision in the same account.

Cost Consideration: OAM cross-account data sharing incurs no additional charges for shared logs and metrics. However, each monitoring account will independently incur standard CloudWatch costs for alarms and dashboards. Plan your alarm distribution across monitoring accounts accordingly. For information about pricing, See Amazon CloudWatch pricing.

Recommendation 5: Monitor additional service quotas in the central account

The Metrics Insights 200-alarm quota isn't the only quota to watch. A centralized monitoring account can trigger other quotas. For a list of CloudWatch quotas, see CloudWatch service quotas.

The following table summarizes the main quotas to monitor, their associated risks, and recommended mitigations:

ResourceQuotaRiskMitigation
Metric Insights alarm200 (non-adjustable)MEDIUMFollow above recommendations
Dashboard widgets500 for each dashboardMEDIUMSplit into dashboards for each team
OAM links for each source account5 (non-adjustable)MEDIUMKeep monitoring account segmentation to less than or equal to 5

Recommendation: Set up service quota alarms in the centralized monitoring account to notify you before you reach the quotas. To set up alarms for quotas, see Visualizing your service quotas and setting alarms and Service Quotas and Amazon CloudWatch alarms. You can also use Service Quotas Automatic Management to proactively monitor quotas and receive notifications.

Conclusion

In this article, we explored the challenge of CloudWatch Metrics Insights alarm quotas at scale and recommended an architectural approach using centralized cross-account monitoring that combines the following components:

  • Metrics Insights query consolidation to monitor entire fleets with one alarm.
  • Composite alarm hierarchies to reduce the number of notifications without using the Metric Insights quota.
  • CloudWatch centralization rules for cross-account, cross-Region telemetry replication, the preferred model for most customers.
  • Environment-segmented monitoring accounts for production, non-production, and security to multiply your available quota.
  • CloudWatch OAM for cross-account trace visibility and query federation.

Organizations can effectively manage alarm quota constraints while improving operational visibility at scale. This approach treats monitoring as a first-class design concern rather than an afterthought. When you plan your observability architecture alongside your workload architecture, you avoid reaching your quotas and build a foundation that scales with your AWS infrastructure.

For guidance on how to implement this architecture in your environment, customers with Enterprise or Business Support plans can engage their TAM to review alarm architecture and quota planning. TAMs routinely help customers design monitoring account structures, right-size alarm strategies, and proactively manage service quotas. To learn more, see AWS Support plans. For immediate assistance, contact Support.

About the authors

Himanshu Dewan

Himanshu Dewan

Himanshu is a Delivery Consultant with AWS ProServe. He works with enterprise customers to design and implement cloud-native solutions at scale, including building CI/CD pipelines, scaling containerized platforms on Amazon EKS, and architecting secure, Well-Architected infrastructure on AWS.

Ashish Kumar

Ashish Kumar

Ashish Kumar is a Cloud Support Eng. II with AWS Premium Support team. He works with customers to investigate issues related to AWS observability and Networking at AWS. He draws on over 6 years of industry experience to help customers troubleshoot complex issues and design resilient, secure network architectures. Outside of work, Ashish enjoys reading, music and exploring new destinations.