AWS Bedrock : Nova Pro with high concurrent traffic

For your high-traffic scenario requiring 2,100 requests per minute with Nova Pro in Frankfurt, you're facing some challenges since the default quota for Nova Pro in Frankfurt is only 50 requests per minute (RPM).

Here are your options to handle this high concurrent traffic:

1. Request a Quota Increase: Unfortunately, according to available information, the on-demand model inference requests per minute for Amazon Nova Pro is marked as not adjustable. This means you cannot simply request a higher RPM limit through the standard Service Quotas process.

2. Cross-Region Inference: You mentioned you've already enabled cross-region inference, which is a good step. This feature distributes traffic across multiple AWS Regions, increasing your overall throughput beyond single-region limits. Cross-region inference keeps requests within the same geography (e.g., requests made within the EU stay within EU regions) while providing higher throughput than calling a model in a single region.

3. Implement Request Queuing: Consider implementing a queuing mechanism between your application and the model to manage the flow of requests. This can help prevent dropped requests due to throughput constraints and is particularly effective for architectures with high demand. This approach follows event-driven messaging patterns and can make your architecture more robust.

4. Contact AWS Support Directly: For quota increases that aren't available through the standard Service Quotas console, you should contact AWS Support directly. Select the appropriate category when opening a support case and clearly explain your use case and throughput requirements. Note that due to high demand, AWS prioritizes customers who are already consuming their existing quota allocation.

5. Consider Architecture Optimization: Review your application architecture to see if you can optimize how you're using the model. This might include batching requests where possible or implementing client-side rate limiting.

Given that provisioned throughput isn't available in Frankfurt and the standard quota is significantly lower than your target of 2,100 RPM, your best immediate option is to maximize the benefits of cross-region inference while working directly with AWS Support to explore potential special accommodations for your use case.
Sources
Bedrock Quota For Nova Pro | AWS re:Post
Unlocking the Potential of AWS Bedrock: Understanding Customization, Throughput, and Pricing | AWS re:Post
Increase throughput with cross-Region inference - Amazon Bedrock
GENREL01-BP01 Scale and balance foundation model throughput as a function of utilization - AWS Well-Architected
How to increase request rate in AWS Bedrock | AWS re:Post