Monitoring 5000 API Gateway + DynamoDB Resources for 10M Invocations — Metrics & Cost Strategy

To track and alert on the combined invocations across 10,000 resources (5,000 API Gateway endpoints and 5,000 DynamoDB tables), while keeping it scalable and cost-efficient, here are some insights and solutions:

1. AWS-Native Approaches CloudWatch Custom Metrics (Individual Metrics): Issue: Tracking individual metrics for 10,000 resources means you’ll have 10,000 custom metrics, which can quickly become costly.

Cost: AWS provides 3 free custom metrics per account, and additional metrics cost $0.30 per metric per month. If you track each API Gateway and DynamoDB table individually, that’s 10,000 metrics, which would cost you approximately:

10,000 metrics x $0.30/metric = $3,000/month (for all custom metrics).

CloudWatch Metric Math: Best Approach: Use CloudWatch Metric Math to aggregate metrics into one single custom metric.

This allows you to define a mathematical expression that combines the invocation metrics from API Gateway and DynamoDB, giving you a total invocation count. You can aggregate API Gateway and DynamoDB metrics into a single custom metric and then create an alarm based on that metric. You won’t need 10,000 individual metrics—just one aggregated metric.

Cost: The cost of using Metric Math is part of the custom metric charges, but instead of tracking 10,000 custom metrics, you only track one, making it much more cost-effective.

Lambda Roll-Up (Aggregation): Approach: Set up a Lambda function that aggregates the invocation counts from API Gateway and DynamoDB at regular intervals (e.g., every minute or hour) and then sends the total invocation count as a single custom metric to CloudWatch.

Cost: You only pay for the Lambda invocation (which is very cheap for periodic aggregations) and for the custom metric that Lambda sends to CloudWatch. Lambda pricing starts at $0.20 per 1 million invocations. Custom metrics from Lambda cost $0.30 per metric.

You’d essentially end up with a single custom metric reporting the total invocations, which keeps your costs low.

Embedded Metric Format (EMF): Approach: If you already use Lambda or other services that emit logs, consider using Embedded Metric Format (EMF) to embed CloudWatch custom metrics directly into your logs. This can be a cost-efficient way to track aggregate data over time.

Instead of creating separate custom metrics for each resource, you can aggregate the invocation counts and publish them as custom metrics via your Lambda logs or other services.

Cost: Similar to Lambda, you only pay for the logs and the custom metrics generated from them.

Contributor Insights: Not Ideal for Your Use Case: Contributor Insights is typically used for analyzing high-cardinality data to understand trends and behavior in logs (e.g., identifying top contributors to API Gateway or DynamoDB). It’s not designed for real-time metric aggregation or alerting based on a total count, so it may not fit your needs here.

2. CloudWatch Alarms and Aggregation Single CloudWatch Alarm: To create a single CloudWatch alarm based on the aggregate invocation count, use Metric Math or a Lambda roll-up to create a single custom metric. Then, set an alarm based on this metric that triggers when it exceeds 10 million.

Example Metric Math expression: Sum([m1, m2, m3, ..., m10000]) This expression can aggregate metrics from 5,000 API Gateway and 5,000 DynamoDB resources into one total.

Delay Tolerance: Since slight delays (5-15 minutes) are acceptable, Metric Math or Lambda aggregation will work fine. Lambda can aggregate data every minute or longer and push the result to CloudWatch at a scheduled interval.

3. Why Only 500 API Gateway in Automatic Dashboards? Limitations: CloudWatch dashboards have a default limit of 500 resources per API Gateway in the automatic dashboard. However, this is just a dashboard limitation and doesn’t affect how you set up custom metrics or alarms.

Solution: If you want to create dashboards for all 10,000 resources, you’ll need to manually create a custom dashboard. CloudWatch dashboards can support more than 500 widgets, but you may need to group your resources logically (e.g., by region, by API Gateway stages, etc.).

4. Cost Estimation Let’s break down the potential costs for each approach:

Custom Metrics (Tracking individually):

10,000 metrics x $0.30/metric = $3,000/month Metric Math or Lambda-based Roll-up (Tracking as a single metric): 1 custom metric x $0.30/metric = $0.30/month Lambda invocations: If you roll up data every minute, you'd have 43,200 invocations per month (30 days x 24 hours x 60 minutes). Lambda cost = $0.20 per 1M invocations = $0.00864/month for 43,200 invocations. Total cost = $0.30 + $0.00864 = $0.30864/month.

DynamoDB Metrics: DynamoDB also provides free metrics at the service level (e.g., read/write capacity, throttling), but if you want to track invocations specifically, you'd need to aggregate them manually.

5. AWS Best Practices Cost Efficiency: Use CloudWatch Metric Math or Lambda to aggregate metrics into one custom metric. This minimizes the number of metrics and reduces costs significantly.

Scalability: Both Metric Math and Lambda aggregation are highly scalable and handle large numbers of resources with minimal overhead. Alerting: Create a CloudWatch alarm based on the aggregated custom metric to alert you when the 10 million invocation threshold is exceeded.

regards, M Zubair https://zeonedge.com

Math function only work for aggregation of 10 metrics as AWS document. If I call the api to get the dynamodb WCU &WCU METRIC it will also cost me?