By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

What tokenizer do the Titan Text models use?

0

I would like to calculate the number of tokens before sending a prompt to the Titan Text LMM service. What tokenizer do the Titan Text models use, and how can I run it locally?

Topics
Machine Learning & AIGenerative AI on AWS
Tags
Amazon Bedrock
Language
English
Ben
asked 2 months ago664 views
2 Answers
0

I am not finding it published in documentation, but if you ask Amazon Q in the AWS console, you get responses referencing WordPiece and SentencePiece tokenizers.

profile pictureAWS
EXPERT
iBehr
answered 2 months ago
  • Ben
    2 months ago

    Thanks for searching, I haven't accepted the answer yet as I would ideally like to know the exact tokenizer. However, WordPiece and SentencePiece might be able to be used as estimates

0

To calculate the number of tokens for a given prompt before sending it to a Titan Text-like Large Language Model (LMM) service, you would typically need to use the same tokenizer or a very similar one. If the exact tokenizer used by Titan Text is publicly available, you can run it locally by installing the necessary library and using the tokenizer to process your text.

profile picture
EXPERT
Osvaldo Marte
answered 2 months ago
  • Ben
    2 months ago

    Yes, but you've just added extra context to the question and not really answered it I'm afraid. I'm looking for the exact tokenizer. Since the use of the LLMs is partly charged per input token I would expect to have the ability to calculate this before sending the prompt

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content