Skip to content

Eight anti-patterns to avoid when running Amazon Bedrock at scale

10 minute read
Content level: Intermediate
0

Amazon Bedrock cross-Region inference raises throughput, but on its own it doesn't make an application resilient. This article is for builders and architects running Bedrock inference in production. It explains eight common design and monitoring mistakes, why each matters, and the fix to build into the client layer, so teams can close resilience gaps before they cause an outage.

Introduction

Design choices that work during an Amazon Bedrock pilot can cause problems under production traffic. Many are easy to miss until traffic becomes high, spiky, or shared across teams.

This article describes eight common anti-patterns, ordered from broad architecture risks to narrower monitoring mistakes. Each section starts with the fix, then explains what the anti-pattern looks like, why it matters, and how to apply the fix. The guidance applies whether applications call Bedrock directly, use a shared client library, or go through an LLM gateway.

In this article, the client layer means whatever sits between your application and Bedrock: a gateway, a shared library, or the application itself. Most of the fixes below are built there.

This article focuses on on-demand inference through bedrock-runtime and bedrock-mantle. Provisioned Throughput and batch inference use different capacity models and are outside its scope.

First: what cross-Region inference is for

Cross-Region inference (CRIS) lets Amazon Bedrock run your request in another AWS Region. A geographic profile, such as EU, keeps requests inside that geography. A global profile can use any supported commercial Region. Bedrock picks the destination for you. This raises throughput and reduces throttling, which makes CRIS a capacity feature. It is not a failover or disaster-recovery design, and it does not protect against every model, provider or Regional disruption. If your application must keep working through a Regional impairment, build and test failover in the client layer.

See Route model inference requests across AWS Regions with cross-Region inference and Implementing resilience patterns with Amazon Bedrock and LLM gateway.

Most of the patterns below follow from this.

1. One model, with nothing to fall back to

The fix: fail over across profiles, then models, using a circuit breaker.

What it looks like: every request uses the same direct model ID or the same inference profile.

Impact: the application has no controlled alternative when it encounters sustained capacity errors or a model disruption.

Details: build an ordered fallback chain in the client layer. Step through the inference profiles available from your source Region and allowed by your data-residency policy, for example a global profile and an eligible geographic profile. Don't assume a profile for another geography can be called from your source Region. If those profiles stay constrained, step down to a second model, then to an older-generation model with its own quotas. Try every eligible profile for the stronger model before you use a weaker one. Decide per workload whether a weaker answer is better than a clear error. Failover can also reduce prompt-cache hits, so input cost and latency rise until the cache warms up. Don't switch per request. Use a circuit breaker instead. Switch when errors pass a threshold, stay switched, and send a small share of traffic back to the first choice now and then to see if it has recovered. Let in-flight requests finish so the next pool takes load gradually.

Check data residency before you design the chain. If a workload must stay in one geography, its chain can only use profiles for that geography, and more of its resilience must come from the model ladder.

2. Retrying and hoping

The fix: retry a few times, then step to a different profile or model.

What it looks like: the plan for capacity errors is "retry with backoff".

Impact: a retry goes back to the same profile, so it might return to the same constrained destination. Retrying does not create an application-controlled fallback. Depending on the SDK and its retry configuration, throttled requests might be retried before the application receives an error, increasing its end-to-end response time.

Details: treat a retry as a short pause, not a recovery plan. After a small number of attempts, the client layer should step to a different profile or model (see #1). Use AWS CloudTrail to check which Region your requests run in (additionalEventData.inferenceRegion) and check each step of your chain the same way. Two profiles that look different can run in the same Region.

AWS SDKs change their default retry behaviour in November 2026. Retries of transient errors such as 5xx start sooner, and the SDK stops retrying sooner when failures persist. Test your fallback chain with the new defaults, because a failing pool will hand control back to your code faster.

See Announcing updated retry behavior for AWS SDKs and Tools.

3. All workloads sharing one account, one Region and one model

The fix: separate workloads by business impact, with their own limits or accounts.

What it looks like: production systems, internal tools, AI agents, CI and offline jobs all call the same model through the same inference option (direct, geographic CRIS or global CRIS) from the same account and source Region.

Impact: they share the quota allocation for that model and inference option. A large offline job can use up the quota and slow down or block the production system next to it.

Details: if you use a gateway, use it to enforce per-tier concurrency, token budgets and queue limits. Without a gateway, nothing shared enforces these limits, so separate accounts become the main way to isolate quotas. Separate accounts add governance work, but create quota and fault-isolation boundaries and clearer cost attribution, even with consolidated billing. Roll out changes to the lowest-risk tier first and to production last.

4. A gateway in one Region only

The fix: run the gateway in at least two application-serving Regions.

What it looks like: a single gateway deployment handles calls to Bedrock, including profiles that might route inference across several destination Regions.

Impact: the gateway provides one place to control routing, but it also becomes part of the application's availability path. If the gateway or one of its Regional dependencies becomes unavailable, the cross-Region capacity behind it does not keep the application working.

Details: deploy the gateway across at least two application-serving Regions when it must survive a Regional impairment. Add health-aware routing and make its state, configuration and dependencies resilient as well. You do not need to deploy the gateway in every CRIS destination Region; Amazon Bedrock performs that destination routing.

If some teams use the gateway while others call Bedrock directly, remember that gateway controls cannot observe or protect the traffic that bypasses them.

5. Sizing quota for one endpoint when you call two

The fix: size and monitor quotas on every endpoint you call.

What it looks like: a team sizes and monitors its quota on bedrock-runtime, but some workloads also call bedrock-mantle, the OpenAI-compatible endpoint.

Impact: the two endpoints use independent capacity controls. bedrock-runtime uses per-model token quotas and, for some models, requests-per-minute quotas. On bedrock-mantle, models with published quotas have separate input- and output-token quotas. Other models might not have per-account quotas exposed in Service Quotas. Increasing an allocation for one endpoint does not increase capacity on the other.

Details: identify the endpoint, model or inference profile, and source Region used by each workload. Review the allocations exposed for that exact combination and request increases before production depends on them. For bedrock-mantle models without published per-account quotas, throughput depends on capacity AWS manages, so handle transient throttling with backoff and contact AWS Support if throttling affects production. AWS recommends bedrock-runtime for new applications. If you use bedrock-mantle for a feature only it offers, size and monitor it as a separate service.

See Quotas for the bedrock-runtime endpoint and Quotas for the bedrock-mantle endpoint.

6. Reading latency on its own

The fix: read latency alongside OTPS, TimeToFirstToken, input and output tokens, and throttles.

What it looks like: a dashboard shows InvocationLatency increasing, and the team assumes that model generation has slowed.

Impact: InvocationLatency includes the time to produce the first token and the time spent generating the rest of the response. It can increase because prefill (the time the model spends reading the prompt before it writes the first token) takes longer, the response contains more output tokens, or the model generates tokens more slowly. A latency chart alone cannot distinguish between these causes.

Details: chart output tokens per second (OTPS) next to InvocationLatency, TimeToFirstToken, InputTokenCount, OutputTokenCount and InvocationThrottles. When latency rises, check in this order:

  1. Is InvocationThrottles above zero? Retries after throttling can make your application slower overall, but they don't increase InvocationLatency. Check quota and your client-side latency separately.
  2. Has OTPS dropped? The model is generating tokens more slowly. Contact AWS Support.
  3. Is OTPS steady but TimeToFirstToken up? Check InputTokenCount. If it rose, prompts got longer. If it's steady, prefill got slower, so check the cache-hit ratio.
  4. Is OTPS steady but OutputTokenCount up? Responses got longer. Check maxTokens and any recent change to the model or prompt.

OTPS requires TimeToFirstToken, so it applies only to streaming requests made through ConverseStream or InvokeModelWithResponseStream. For non-streaming requests, compare InvocationLatency with InputTokenCount and OutputTokenCount instead. If latency rises while both token counts stay steady, contact AWS Support.

For a five-minute CloudWatch view, use the p50 statistic for all three input metrics and calculate:

OutputTokenCount / (InvocationLatency - TimeToFirstToken) x 1000

See Diagnose InvocationLatency increases using output tokens per second (OTPS).

7. A dashboard built on one namespace

The fix: monitor both the AWS/Bedrock and AWS/BedrockMantle namespaces.

What it looks like: all Bedrock dashboards and alarms read the AWS/Bedrock CloudWatch namespace.

Impact: traffic on bedrock-mantle publishes to a separate namespace, AWS/BedrockMantle, with different metric names and fewer metrics. A dashboard built on AWS/Bedrock leaves that traffic out without any warning.

Details: check which namespace each workload reports to. Where CloudWatch doesn't publish a metric you need, such as latency on bedrock-mantle, have the client layer measure and publish it.

See Monitor bedrock-mantle inference using CloudWatch metrics.

8. Adding server errors to the denominator

The fix: divide server errors by Invocations.

What it looks like: server-error rate is calculated as InvocationServerErrors ÷ (Invocations + InvocationServerErrors).

Impact: Invocations already includes requests that Bedrock accepted for processing and that later failed with a server-side or client-side error. Adding InvocationServerErrors again counts server-error requests twice and understates the rate.

Details: calculate:

InvocationServerErrors ÷ Invocations

Label this specifically as the server-error rate. Chart InvocationClientErrors and InvocationThrottles separately so that each failure class remains visible. Throttled requests are not included in Invocations.

See Monitor bedrock-runtime inference using CloudWatch metrics.

A short checklist

  • Record how each workload calls Bedrock and which endpoint it uses.
  • Check quotas and headroom for every model and profile in the fallback chain.
  • Use CloudTrail to verify where cross-Region requests run.
  • Build and test fallback across eligible profiles and supported models.
  • Isolate critical workloads and make the client layer resilient.
  • Monitor OTPS, throttles, client errors and server-error rate.