ChatBedrockConverse.with_structured_output not working with meta.llama3-2-11b-instruct-v1:0 model

I apologize, but it appears that the issue you're experiencing is not directly related to the Meta Llama 3.2 11B model's capabilities, but rather to how the ChatBedrockConverse class is interacting with the model through Amazon Bedrock's API.

The error message you're receiving suggests that there's a problem with the parameter validation, specifically with the toolConfig.tools[0].toolSpec.description parameter. This parameter is expected to have a minimum length of 1, but it's being sent with a length of 0.

Unfortunately, the structured output functionality you're trying to use (with_structured_output) may not be fully compatible with the Meta Llama models on Amazon Bedrock in the way you're attempting to use it. The Converse API for Meta Llama models doesn't natively support tool configurations in the same way some other models might.

As an alternative approach, you could try the following:

1. Use the basic Converse API without the structured output wrapper.
2. In your prompt, explicitly describe the structure you want the output in (e.g., JSON format matching your ImageMetadata class).
3. Parse the response manually or use a JSON parser to convert it to your desired structure.

Here's a basic example of how you might structure your request:

import boto3
from botocore.exceptions import ClientError

client = boto3.client("bedrock-runtime", region_name="your-region")
model_id = "meta.llama3-2-11b-instruct-v1:0"

user_message = """
Please provide information about an image in the following JSON format:
{
  "title": "string",
  "description": "string",
  "tags": ["string", "string", ...]
}
Describe a sunset over the ocean.
"""

conversation = [
    {
        "role": "user",
        "content": [{"text": user_message}],
    }
]

try:
    response = client.converse(
        modelId=model_id,
        messages=conversation,
        inferenceConfig={"maxTokens": 512, "temperature": 0, "topP": 0.9},
    )
    
    response_text = response["output"]["message"]["content"][0]["text"]
    print(response_text)
    # You can then parse this response_text as JSON and convert it to your ImageMetadata object

except ClientError as e:
    print(f"Error: {e}")

This approach, while requiring more manual work, should be more reliable and compatible with the Meta Llama models on Amazon Bedrock. You'll need to ensure your prompt clearly specifies the desired output format, and then parse the response accordingly.
Sources
Invoke Meta Llama on Amazon Bedrock using Bedrock's Converse API - Amazon Bedrock
Invoke Meta Llama on Amazon Bedrock using Bedrock's Converse API - AWS SDK Code Examples
Meta Llama - Models in Amazon Bedrock - AWS

Thanks! Explicitly adding the JSON format works but using PydanticOutputParser.get_format_instructions doesn't with the rest of the user prompt being the same. I'm guessing that is a Meta Llama problem.