By Kiran Khambete and Krish Balaraman

Building OTPS correctly: four traps in Bedrock's latency SLO method

Read these first

This article assumes you've read three AWS sources:

• Diagnose InvocationLatency increases using output tokens per second (OTPS) — the OTPS formula, the metric math expression, and two alarm patterns.
• Optimize your applications for scale and reliability on Amazon Bedrock — 429 handling, the max_tokens quota reservation, the three-stage token lifecycle, and quota sharing across applications. Its burndown rates are older than the user guide's.
• How tokens are counted in Amazon Bedrock — current burndown rates and worked scenarios. One catch: Scenario 2 calls its optimized request "only slightly higher" at admission than at settlement, but the numbers are 5,250 and 9,000. Admission counts max_tokens at face value; settlement applies the burndown rate to the actual output. On any model above 1:1, a right-sized max_tokens can leave admission below the settled figure.

This covers four ways the OTPS method breaks on real traffic, a diagnostic for calls that don't stream, and a runbook.

The method

Bedrock publishes InvocationLatency (request to last token) and TimeToFirstToken (request to first token, streaming only) in the AWS/Bedrock namespace (runtime metrics). Subtract one from the other and you have generation time. Divide output tokens by that and you have OTPS:

OTPS = OutputTokenCount / (InvocationLatency - TimeToFirstToken) * 1000

OTPS isolates service-side throughput, so a longer prompt or response doesn't trip it. As CloudWatch metric math — m1 latency, m2 output tokens, m3 TTFT, each at p50 over 5 minutes:

m2 / (m1 - m3) * 1000

Trap 1: a mixed identifier gives a biased number, not an error

TimeToFirstToken is published only for ConverseStream and InvokeModelWithResponseStream. InvocationLatency and OutputTokenCount are published for all four invocation operations: Converse, ConverseStream, InvokeModel, InvokeModelWithResponseStream (AWS ML blog). There's no operation dimension — just ModelId (runtime metrics).

So if one identifier serves both streaming and non-streaming traffic, the expression still returns a number. It subtracts a streaming-only TTFT median from a latency median that includes non-streaming calls. The result looks fine and isn't.

Worse, it tracks your traffic mix. Shift calls between streaming and non-streaming and OTPS moves even though the model's speed hasn't changed — a workload shift read as a throughput change, the exact thing OTPS is supposed to rule out.

Fix: keep streaming and non-streaming traffic on separate identifiers, confirm the ModelId values differ in your account, and baseline on the streaming identifier only. The OTPS page says its procedures "apply only to traffic on those operations" but doesn't tell you how to enforce it — and CloudWatch won't warn you if you don't.

Trap 2: one model is several series

ModelId is the identifier you called, not the underlying model. On a geographic or global cross-Region profile, "the ModelId dimension corresponds to your inference profile identifier" (AWS ML blog).

So the same model gives separate series for an in-Region call, a us. profile, and a global. profile. Latency-optimized requests are separate again — note the wording: "you can also view metrics for latency optimized requests in Amazon CloudWatch logs under "model-id+latency-optimized"" (latency-optimized inference). Logs, not metrics. Confirm what shows up in your own account before building a widget on it.

Fix: build per identifier you actually call, and use the same identifier in all three OTPS terms. A profile ID in one term and a bare model ID in another gives a number with no meaning, and no error either.

Trap 3: OTPS can't be split by service tier

service_tier is a request, not a guarantee. ResolvedServiceTier "shows the actual tier that served your requests" (service tiers) — a latency percentile across tiers blends two service levels.

But the tier dimensions are documented on TimeToFirstToken and EstimatedTPMQuotaUsage only (AWS ML blog). Neither that post nor the service-tiers page lists them on InvocationLatency or OutputTokenCount, and OTPS needs both.

So you can split TTFT by tier; you can't split OTPS. Split TimeToFirstToken by ResolvedServiceTier, and say in your SLO document that OTPS is blended across tiers.

Counting traffic served below the requested tier is harder still — Invocations isn't documented with the tier dimension. Two routes:

• SampleCount of TimeToFirstToken grouped by ResolvedServiceTier. No code, streaming only.
• Record the served tier per request and publish your own metric. It "is visible in API response and AWS CloudTrail Events" (service tiers). Covers everything, needs instrumentation.

Trap 4: p50 OTPS isn't the median request's throughput

The expression computes p50(OutputTokenCount) / (p50(InvocationLatency) − p50(TimeToFirstToken)) × 1000 — a ratio of three separate medians, not the median of per-request throughput. CloudWatch never joins the three values per request, so nothing gives you the latter.

Treat OTPS as a trend, baseline and alarm input, not as the throughput of any one request.

When calls don't stream: a three-metric diagnostic

TimeToFirstToken isn't published for Converse or InvokeModel, so OTPS isn't available.

For non-streaming calls the user sees nothing until the last token, so InvocationLatency is the Bedrock portion of that wait — the right Bedrock metric here, not a fallback. It's still measured server-side, so your SLO should still use client-side end-to-end timing (see the worksheet below).

InvocationLatency bundles prompt processing, which "scales primarily with input length," and token generation, which scales with output (OTPS). A rise can mean a longer prompt, a longer response, or a slower model. Read three metrics together:

InvocationLatency	OutputTokenCount	InputTokenCount	Likely cause
↑	↑	any	Longer responses — prompt change, higher max_tokens, model update
↑	flat	↑	Longer prompts — new system prompt, more retrieval context. Not an incident
↑	flat or ↓	flat	Prompt processing or generation slowed — service-side
flat	↑	flat	Throughput improved, or output grew within a stable latency

Two checks first:

• Cache hits. InputTokenCount excludes cached tokens (token counting). A drop in cache hit rate reprocesses more of the prompt and raises latency with no change in InputTokenCount. Check CacheReadInputTokenCount before calling a flat-input row service-side.
• Throttling. Throttled requests "don't count as either Invocations or Errors" (runtime metrics), so retry delay never shows in InvocationLatency. Check InvocationThrottles separately, always.

This is weaker than OTPS — trends instead of one alarmed number — but it's what you have. The metric definitions are AWS's; the diagnostic is mine.

Runbook

TTFT	OTPS	InvocationLatency	Likely cause	First action
↑	flat	↑	Longer input: prompt, retrieval context, system prompt	Check recent deploys and retrieval config. Not a service incident
flat	↓	↑	Service-side throughput change	Check the AWS Health Dashboard, then open a support case with request IDs and UTC timestamps (scaling best practices)
flat	flat	↑	Longer output: more verbose responses, higher max_tokens, model change	Check OutputTokenCount and inference parameters
↑	↓	↑	Both, often under load	Check InvocationThrottles, then the quota sources listed at the top
flat	flat	flat, users still complain	Latency is outside Bedrock	Instrument retrieval, Guardrails and your own code

For the last row:

• Guardrails publish InvocationLatency in AWS/Bedrock/Guardrails, split by an Input/Output content-source dimension (Guardrails metrics). A dashboard scoped to AWS/Bedrock won't show them.
• Cross-Region serving Region isn't in these metrics at all. CloudTrail logs cross-Region calls in the source Region, and "the additionalEventData.inferenceRegion field records where each request was processed" (AWS ML blog, New Zealand Region; also Securing Amazon Bedrock cross-Region inference) — documented in AWS blogs, not the user guide.

Alarming on OTPS

OTPS is an expression, so you alarm on it with a metric math alarm. The OTPS page gives two patterns:

• Static threshold at roughly 80 percent of a measured baseline. AWS's example: a 55 tok/s baseline gives a threshold of 44.
• Anomaly detection, "lower than the band," at 2–3 standard deviations. Needs at least two weeks of history.

Pick the missing-data setting on purpose; the four behave differently (missing data):

Setting	Behavior
breaching	Gaps count as bad
notBreaching	Gaps count as good
ignore	Current alarm state is maintained
missing (default)	If every datapoint in the range is missing, the alarm goes to INSUFFICIENT_DATA

For OTPS, a gap usually means no streaming traffic, not degradation. The code below uses ignore, which holds the last state through a lull — so an alarm already in ALARM stays there until data returns. Use missing if quiet periods should read as unknown. To page on a traffic stop specifically, alarm on Invocations instead of setting breaching here.

In the console

For one alarm, the console is enough. Steps, per the OTPS page:

1. In CloudWatch: Metrics → All metrics → search Bedrock → By ModelId.
2. Select InvocationLatency, OutputTokenCount and TimeToFirstToken for one streaming-only identifier.
3. Set each to statistic p50, period 5 minutes.
4. Add math → Start with empty expression → enter m2 / (m1 - m3) * 1000, label it OTPS. Check that m1, m2, m3 match the IDs the console assigned to latency, output tokens and TTFT.
5. Click the alarm icon on the OTPS expression. Threshold type: Static. Condition: Lower than. Enter your threshold.
6. Under additional configuration, set 3 of 5 datapoints and pick the missing-data treatment.
7. Pick your SNS topic, create the alarm.

As code

Reach for code when you need the same alarm across several identifiers, Regions or accounts, or when you'll re-baseline after every model update. It also forces all three metrics onto one identifier, which sidesteps the Trap 2 mismatch. Run it once, from anywhere with cloudwatch:PutMetricAlarm, in the Region your application calls Bedrock from — the source Region, for cross-Region inference. CloudWatch takes over evaluating it from there. The same alarm works in CloudFormation, CDK or Terraform.

import boto3

cw = boto3.client("cloudwatch", region_name="us-east-1")

# Use a streaming-only identifier (Trap 1).
MODEL_ID = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"

# 44 is the AWS example (80% of a 55 tok/s baseline).
# Replace with your own measured baseline.
OTPS_THRESHOLD = 44
SNS_TOPIC = "arn:aws:sns:us-east-1:123456789012:genai-oncall"


def metric(mid, name):
    return {
        "Id": mid,
        "MetricStat": {
            "Metric": {
                "Namespace": "AWS/Bedrock",
                "MetricName": name,
                "Dimensions": [{"Name": "ModelId", "Value": MODEL_ID}],
            },
            "Period": 300,
            "Stat": "p50",
        },
        "ReturnData": False,
    }


cw.put_metric_alarm(
    AlarmName="Bedrock-OTPS-Low",
    AlarmDescription=(
        "OTPS below baseline: model-side throughput degradation, "
        "not a workload change."
    ),
    Metrics=[
        metric("m1", "InvocationLatency"),
        metric("m2", "OutputTokenCount"),
        metric("m3", "TimeToFirstToken"),
        {
            "Id": "otps",
            "Expression": "m2 / (m1 - m3) * 1000",
            "Label": "OTPS",
            "ReturnData": True,
        },
    ],
    ComparisonOperator="LessThanThreshold",
    Threshold=OTPS_THRESHOLD,
    EvaluationPeriods=5,
    DatapointsToAlarm=3,
    TreatMissingData="ignore",
    AlarmActions=[SNS_TOPIC],
)

Adapted from the AWS example.

Latency budget worksheet

Millisecond figures vary too much by model, prompt and config to publish. What's worth keeping is the instrument for each component.

Component	How to measure	In AWS/Bedrock?
Client to your API	Your request timing	No
Retrieval	Time the vector store call directly	No
Guardrails	InvocationLatency in AWS/Bedrock/Guardrails, split by Input/Output (Guardrails metrics)	No, separate namespace
Prompt processing	TimeToFirstToken (streaming only)	Yes
Token generation	InvocationLatency − TimeToFirstToken; normalize with OTPS	Yes
Your post-processing	Your own timing	No
End-to-end	Client-side timing, published as a custom metric	No — this is the SLO number

TTFT is measured server-side, so it "reflects actual service-side latency without noise from network conditions or client-side processing" (AWS ML blog) — which is also why it can't substitute for client-side end-to-end timing. Measure the total at the client. Don't add component p99s: percentiles don't sum, and the total describes a request that almost never happens.

Limitations

• Streaming-only identifiers. Everything here rests on Trap 1. On mixed traffic, OTPS is wrong and nothing flags it.
• Bedrock only. These metrics cover the Bedrock call, not your whole request path.
• Successful requests only. TimeToFirstToken and EstimatedTPMQuotaUsage are scoped to "Successfully completed requests" (AWS ML blog). Throttle counts "will depend on your retry settings in the SDK" (runtime metrics). A client stuck in retries can wait while your percentiles look fine. Watch InvocationThrottles alongside latency.
• Model updates. A threshold set for one model version doesn't carry to the next. Version it.
• Cross-Region inference. Metrics attribute to the inference profile, not the destination Region (AWS ML blog). Use CloudTrail's inferenceRegion field for the serving Region.
• bedrock-mantle. Out of scope. It publishes to AWS/BedrockMantle, and "InvocationLatency and TimeToFirstToken equivalents are not yet published" (mantle metrics). Its metrics also aren't aggregated across Regions the way CRIS traffic is on bedrock-runtime.

Checklist

1. Streaming and non-streaming traffic on separate identifiers; ModelId values confirmed to differ.
2. OTPS graphed per streaming-only identifier, p50, 5-minute period.
3. Same identifier in all three OTPS terms.
4. OTPS alarm on a measured baseline, 3 of 5 datapoints, missing-data setting chosen deliberately.
5. InvocationThrottles, InvocationClientErrors, InvocationServerErrors as separate series.
6. With service tiers: TimeToFirstToken split by ResolvedServiceTier, OTPS documented as blended.
7. Non-streaming calls covered by InvocationLatency, OutputTokenCount, InputTokenCount, with CacheReadInputTokenCount alongside.
8. Client-side end-to-end latency published as a custom metric.
9. Runbook written and exercised once by on-call.
10. Baseline re-measured after each model version change.

Regulated workloads

Under an operational-resilience regime, a customer-facing generative AI service may fall inside an important business service, and the latency SLO may need to match an impact tolerance. This varies by regulator and by how your firm maps its services — check with compliance rather than inferring it here. What this article gives you is the measurement: what each number is, where it comes from, and what it can't show.

References

• Diagnose InvocationLatency increases using output tokens per second (OTPS)
• Monitor bedrock-runtime inference using CloudWatch metrics
• How tokens are counted in Amazon Bedrock
• Service tiers for optimizing performance and cost
• Optimize model inference for latency
• Monitor Amazon Bedrock Guardrails using CloudWatch metrics
• Monitor bedrock-mantle inference using CloudWatch metrics
• Scaling and throughput best practices
• Configuring how CloudWatch alarms treat missing data
• Optimize your applications for scale and reliability on Amazon Bedrock
• Improve operational visibility for inference workloads on Amazon Bedrock with new CloudWatch metrics for TTFT and Estimated Quota Consumption
• Run Generative AI inference with Amazon Bedrock in Asia Pacific (New Zealand)
• Securing Amazon Bedrock cross-Region inference: Geographic and global
• AWS Health Dashboard