Skip to content

Tokenization

πŸ“ Short definition

Turning text into a sequence of numeric identifiers (tokens) the model can process, using a vocabulary learned during training.

πŸ“– Detailed definition

Text is split into tokens (words, subwords, or characters) with an algorithm learned during trainingβ€”the most common being BPE (Byte Pair Encoding). Each token gets a unique number in the model vocabulary. That number is what the model actually sees.

Example: β€œWhere is Paris?” β†’ [4502, 381, 1920, 30]

Vocabulary size varies by model: Llama 3 uses 128,256 tokens. A rare word may split into several subwords, increasing the token count consumed.

πŸ’‘ Why it matters for on-prem AI

Token count for a text directly sets KV Cache size in VRAM and the cost of Prefill. A 200-page PDF can represent hundreds of thousands of tokens.

⚠️ Common pitfalls

  • The same text yields different token counts per model tokenizerβ€”do not assume β€œ1 page β‰ˆ 500 tokens” is universal.
  • French often uses more tokens than English (more subwords created).

πŸ“š Go deeper

  1. 🧠 The Journey of a Prompt (Step 1: tokenization in action)
  2. πŸ’Ύ KV Cache & Context (why token count affects VRAM)

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.