Send images as Base64 files to AWS BedRock Agent

Same issue here! Did anyone have any luck passing an image to the bedrock agent?

LE: I've read through the documentation like a thousand times, and all references to the sessionState dict point to code interpretation type of usecase

Contains information about the files used by code interpreter

As such I don't think that's the path fwd to be including an image in the prompt for an agent. I'm trying now a different path, whereby I ask the cloude3 model ( via the model API not agent API ) to describe the image with some prompt engineering, and then pass that description into the original prompt.

If anything changes in the agent API to allow this to be better streamlined, please let us know.

It looks like your request is consistent with the API doc and it's that specific files parameter that's failing validation, so I'd suggest to double-check whether you're running the most up-to-date version of boto3 and botocore? Seems like the most likely cause would be an old boto version not supporting that parameter yet.

Generally, I believe you should be on the right approach for passing in the base64 bytes.

As suggested by @Alex_T, I updated my boto3 library and that solved the first error I encountered. Now I got a new error about problem with my prompt or file format... Been thinking if Amazon Agent can really receive images, cause there's nothing on the Documentation that firmly says it.

The script is now like this:

def invoke_agent(agent_id, agent_alias_id, session_id, input_text, image_path = None):
    """
    Sends a prompt for the agent to process and respond to.

    :param agent_id: The unique identifier of the agent to use.
    :param agent_alias_id: The alias of the agent to use.
    :param session_id: The unique identifier of the session. Use the same value across requests
                       to continue the same conversation.
    :param prompt: The prompt that you want Claude to complete.
    :return: Inference response from the model.
    """

    try:
        
        
        image_data = None
        if image_path:
            with open(image_path, "rb") as image_file:
                image_data = base64.b64encode(image_file.read()).decode('utf-8')

        # Note: The execution time depends on the foundation model, complexity of the agent,
        # and the length of the prompt. In some cases, it can take up to a minute or more to
        # generate a response.
        response = agents_runtime_client.invoke_agent(
            agentId=agent_id,
            agentAliasId=agent_alias_id,
            sessionId=session_id,
            inputText=input_text,
            sessionState={
            'files': [
            {
                'name': 'picture',
                'source': {
                    'byteContent': {
                        'data': f"b'{image_data}'",
                        'mediaType': 'image/jpeg'
                    },
                    'sourceType': 'BYTE_CONTENT'
                },
                'useCase': 'CHAT'
            }
        ]
            }
        )
            

        completion = ""

        for event in response.get("completion"):
            chunk = event["chunk"]
            completion = completion + chunk["bytes"].decode()

    except Exception as e:
        print(f"Couldn't invoke agent. {e}")
        raise

    return completion


# Define the chat parameters
agent_alias_id = "XXXXXXXX"
agent_id = "XXXXXXX"
session_id = 'XXXXXXX'
input_text = "Hello ma friend"

But now the error is:

EventStreamError: An error occurred (validationException) when calling the InvokeAgent operation: The overridden prompt that you provided is incorrectly formatted. Check the format for errors, such as invalid JSON, and retry your request.

And it's the same error when I try to upload a file directly into the Agent Test UI in Bedrock, like in the print below:

Does someone have a clue on that?

Has anyone found a solution? I am getting the same error "Check the format for errors, such as invalid JSON," Using the latest boto3 & BotoCore: 1.36.13

You can't currently send image files to a Bedrock Agent. It can take other file types per the code above (ex. XLS, DOC, PDF) but internally the Bedrock Agent framework extracts text and passes it to the LLM model. Even though the model itself may be visually capable, it can't receive images within the context of a Bedrock Agent - for example, if you save a .png file as a .pdf and upload it, the Agent will say it doesn't see anything in the file. To process images, you'll need to use the model through Bedrock directly, for example via the Converse API or the Invoke API.