LLM
π Short definition
AI model trained on very large text corpora, able to understand and generate text.
π Detailed definition
Modern LLMs mostly use the Transformer architecture. Once trained, they mainly serve inference: on-demand text generation from a prompt. Practical local performance depends as much on memory hardware as on model size and quantization.
π‘ Why it matters for on-prem AI
Central stack component: all sizing (RAM/VRAM, throughput, latency) follows from it, especially for local inference β rarely for training from scratch.
β οΈ Common pitfalls
- Assuming a βbigger modelβ is always better for every use case.
- Forgetting performance depends as much on machine memory as on the model.
π Go deeper
Want to see what an LLM does when you talk to it?
- π§ The Journey of a Prompt (tokenisation, prefill, decoding β full cycle)
- LLM inference (day-to-day use of a trained LLM)
- ποΈ 4-bit & 8-bit quantization (shrinking a model to fit memory)
π See also
Phase where an already trained LLM produces a response token by token, in two steps (prefill then decoding); it is what sizes on-premise hardware.Reducing numerical precision of LLM weights to lower memory footprint and speed up inference.Situation where memory throughput limits performance more than raw compute power, especially in autoregressive generation; not to be judged on TFLOPS alone.Architecture that combines document retrieval and LLM generation to ground answers in internal sources.