🧠 The Journey of a Prompt: How Does an LLM Work?
When you type a question into a local AI model, there is an illusion of magic: the machine seems to “understand” the text and “think” about the answer.
In reality, a processor does not understand French—or any human language. It only executes billions of multiplications at blistering speed. To understand why AI demands so much hardware (and why your PC can feel sluggish), you need to lift the hood and see how text is turned into mathematical computation.
Here is the exact journey of your prompt, step by step.
Step 1: Tokenization (The text chopper)
A computer can only compute with numbers. The very first step is therefore to split your sentence into chunks called tokens, then assign a number to each chunk.
Each model has its own dictionary, learned during training. In 2026, a standard model like Llama 3 has a vocabulary of roughly 128,256 possible tokens1. Each number is the exact index of one of those words or subwords.
Step 2: Embedding (The treasure map)
Having the number 4502 for the word “Where” does not help the model grasp its meaning.
The machine therefore converts each number into a long list of mathematical coordinates (a vector). This is called Embedding.
Imagine a 3D map where similar concepts sit close together. The word “King” will be near “Queen” and “Crown.” In a large model (LLM), this “map” is not 3-dimensional but often 4,096 or 8,192 dimensions.
At the end of this step, your short four-word sentence has become a huge grid of thousands of decimal numbers. The real computation can begin.
Step 3: “Prefill” (Ingesting the context)
This is where the inference engine (the software running the AI) launches its heaviest work. The model passes your grid of numbers through dozens of artificial neural network “layers.”
This phase is called Prefill. The model reads your entire prompt at once, in parallel.
The heart of this step is the Attention mechanism: the model cross-relates every word to understand context. It mathematically determines that, in your sentence, the question mark is strongly linked to the word “Where.”
- What happens in the machine: Your chips (CPU or GPU) run at 100% of their compute capacity (TFLOPS) because they can execute all those matrix multiplications simultaneously.
Step 4: KV Cache (Short-term memory)
During Prefill, the model computed the importance and context of each word in your history. If it had to recompute all of that for every new word it generates, it would be painfully slow.
To avoid that, it saves all those intermediate results in your graphics card’s VRAM. This is the famous KV Cache (Key-Value Cache). The longer your prompt (e.g., if you give it a 200-page PDF to read), the larger this KV cache grows and the more it saturates your machine’s memory.
Step 5: “Decoding” (Word-by-word generation)
Now that the model has “digested” your question and stored context in the KV cache, it produces its answer. It does so one token at a time. This is the Decoding phase.
- The model looks at your prompt, looks at its KV cache, and mathematically predicts that the most probable word to start the answer is “Paris”.
- It writes “Paris”.
- The cycle repeats: it takes your prompt + the word “Paris”, re-reads the KV cache, and computes that the next word is “is”.
- It writes “is”.
This loop continues until the model generates a special token that means <END>.
- What happens in the machine: This is a highly sequential phase. The processor must constantly shuttle data to and from memory to fetch model weights and the KV cache, just to generate a single token. That is why generation speed depends on Memory Bandwidth rather than raw compute power alone.
🎯 Summary for the architect
When a user complains that “the AI is slow,” you need to know which phase they mean:
- Takes too long to get started? That is Prefill. The processor lacks raw compute (TFLOPS) to digest the initial prompt, or the context sent is too long.
- Writes the answer too slowly? That is Decoding. The machine hits the “memory wall”: the graphics card memory cannot feed the chip fast enough to compute the next token.
📚 Sources and References
Footnotes
-
Meta, Llama 3.1 Model Card (Architecture, vocabulary of 128,256 tokens, Tiktoken tokenizer), 2024. GitHub — llama-models ↩