"Chat with your document" - Conversational History

Based on my current knowledge, the Anthropic Claude 3 Sonnet model is the only one that currently supports conversational history for the on-the-fly knowledge base integration (i.e., "Chat with your document").

The Haiku model, which uses the retrieve_and_generate API, does not seem to have the same level of support for conversational context as the Claude 3 Sonnet model. The Haiku model appears to generate responses without maintaining the context of the previous conversation.

This difference in conversational history support is likely due to the different architectures and capabilities of the Haiku and Claude 3 Sonnet models. The Claude 3 Sonnet model has been specifically designed and trained to handle more complex conversational interactions, including maintaining context across multiple turns of a dialogue.

It's worth noting that the capabilities of these models may evolve over time, so the current behavior you've observed could change in the future. If you require the ability to maintain conversational history for your use case, the Claude 3 Sonnet model seems to be the best option based on the information available.

All models support conversational history in retrieveAndGenerate API . On the first call to retrieveAndGenerate you get sessionId as part of the response. If you send it to all subsequent requests, the conversation history will be preserved. But "chat with your document" is a different API. Here is Java Example:

public static String documentInsight(String filePath,String modelId,String command) throws IOException {
        var client = BedrockRuntimeClient.builder().region(Region.US_EAST_1).build();

        
        var fileContent = SdkBytes.fromByteArray(Files.readAllBytes(Paths.get(filePath)));

        var textMessage = ContentBlock.fromText(command);
        var document = ContentBlock.fromDocument(doc -> doc.name("document")
                .format(DocumentFormat.PDF)
                .source(DocumentSource.fromBytes(fileContent)));

        var response = client.converse(req -> req
                .modelId(modelId)
                .messages(message -> message
                        .role(ConversationRole.USER)
                        .content(textMessage, document)));

        return response.output().message().content().get(0).text();    
    }