How to attach model inference profile while creating a bedrock agent using python CDK ?

Apparently, claude sonnet 3.5 V2 is only available in us-west-2. So in order to use in us-east-1, you have to use the inference profile is as the model id.

So in my cdk, I replaced model from - anthropic.claude-3-5-sonnet-20241022-v2:0 to us.anthropic.claude-3-5-sonnet-20241022-v2:0

You can find this in Inference and Assessment > Cross-region inference

tl;dr : prefix your model id with "us." to use the cross-regional inference profile instead of the regional foundation model. The cross-region inference profile is preferred (to double your effective quotas) and seems to be required for some newer/larger models.

I just ran into this, and was happy to discover the solution is easier than I expected. The Bedrock user guide says "You can use a cross Region inference profile in place of a foundation model to route requests to multiple Regions." In Bedrock CDK the foundation_model parameter accepts any model identifier: this can be cross-region inference profile id (with us. prefix), an application inference model id, a foundation model id, or ARN of any of these resource types. I would always choose the ID for most API calls, but of course you need the ARN for IAM policies and such.

Also note with the API calls (to Converse, for example) you could pass a model ID or inference profile interchangeably as well.

# your choice of IDs, note the us. prefix on the inference profile 
foundation_model_id = "anthropic.claude-3-5-sonnet-20241022-v2:0"
inference_profile_id = "us.anthropic.claude-3-5-sonnet-20241022-v2:0"
application_profile_id = "your_profile_id"
foundation_model_arn = "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"
inference_profile_arn = "arn:aws:bedrock:us-east-1::inference-profile/us.anthropic.claude-3-5-sonnet-20241022-v2:0"
application_profile_arn = "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/your_profile_id"

support_agent = bedrock.CfnAgent(self, "HorizonIQSupportAgent",
    agent_name="support-agent",
    agent_resource_role_arn=agent_role.attr_arn,
    auto_prepare=True,
    description="support-agent",
    foundation_model=inference_profile_arn,  # Use id or arn for an inference profile or foundation model here
    idle_session_ttl_in_seconds=600,
    instruction=prompt,
    knowledge_bases=[bedrock.CfnAgent.AgentKnowledgeBaseProperty(
        description=kb_description,
        knowledge_base_id=knowledge_base.attr_knowledge_base_id,
    )]
)

and for permissions, if using L1 constructs you'll need to add actions to the inference profile AND to the FMs in each region of the model's cross-region inference. For US endpoints that's often us-east-1 and us-west-2 but some pop up in the smaller regions also. L2 grant_profile_usage construct seems to handle cross-region permissions automatically.

{
  "Effect": "Allow",
  "Action": ["bedrock:InvokeModel*", "bedrock:GetInferenceProfile"],
  "Resource": [
    "arn:aws:bedrock:us-east-1::inference-profile/us.amazon.nova-micro-v1:0",
    "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-micro-v1:0",
    "arn:aws:bedrock:us-west-2::foundation-model/amazon.nova-micro-v1:0",
    "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/your_profile_id"
  ]
}

Hope this helps others on this path.
JR

You can attach the model inference profile by adding an inference_configuration property to your CfnAgent constructor. This property should be set within a PromptOverrideConfiguration object. Here's how you can modify your existing code:

support_agent = bedrock.CfnAgent(self, "HorizonIQSupportAgent",
    agent_name="support-agent",
    agent_resource_role_arn=agent_role.attr_arn,
    auto_prepare=True,
    description="support-agent",
    foundation_model=foundation_model_name,
    idle_session_ttl_in_seconds=600,
    instruction=prompt,
    knowledge_bases=[bedrock.CfnAgent.AgentKnowledgeBaseProperty(
        description=kb_description,
        knowledge_base_id=knowledge_base.attr_knowledge_base_id,
    )],
    prompt_override_configuration=bedrock.CfnAgent.PromptOverrideConfigurationProperty(
        prompt_configurations=[
            bedrock.CfnAgent.PromptConfigurationProperty(
                prompt_type="ORCHESTRATION",
                inference_configuration=bedrock.CfnAgent.InferenceConfigurationProperty(
                    inference_profile_arn="arn:aws:bedrock:us-east-1::inference-profile/us-anthropic-claude-3-sonnet"
                )
            )
        ]
    )
)

In this modified code, we've added a prompt_override_configuration property to the CfnAgent constructor. Within this configuration, we specify a PromptConfigurationProperty with a prompt_type of "ORCHESTRATION" and an inference_configuration that includes the ARN of the inference profile you want to use.

For the US Anthropic Claude 3 Sonnet inference profile, you should use the ARN: "arn:aws:bedrock:us-east-1::inference-profile/us-anthropic-claude-3-sonnet". This is a cross-region inference profile, so you can use either its ARN or its ID.

By adding this configuration, you're telling the agent to use the specified inference profile when invoking the foundation model during the orchestration part of the agent sequence. This should resolve the "cannot invoke with provisioned throughput mode" error you were encountering.

Remember to replace the inference profile ARN if you're using a different profile or if you're in a different region. Also, ensure that your IAM role has the necessary permissions to use the specified inference profile.

Sources
Use an inference profile in model invocation - Amazon Bedrock
CfnAgent — AWS Cloud Development Kit 2.179.0 documentation