Embedding
π Short definition
Coordinate vector encoding the meaning of a token or document in mathematical space β nearby concepts are nearby in that space.
π Detailed definition
After tokenisation, each token becomes a fixed-size vector (e.g. 4096 dimensions for Llama 3.1 8B). In that space, βKingβ and βQueenβ are close; βParisβ and βFranceβ too. That geometry lets the model reason by analogy.
Two distinct uses β do not conflate:
- Internal LLM embeddings: vectors computed at each layer during Prefill/Decoding β they feed attention.
- Search embeddings (RAG): produced by specialized models (e.g.
nomic-embed-text,sentence-transformers) to index documents in a vector database.
π‘ Why it matters for on-prem AI
RAG quality depends heavily on the embedding model chosen. A poor embedding model yields poor retrieval regardless of downstream LLM quality.
β οΈ Common pitfalls
- Internal LLM embeddings and RAG search embeddings are different models for different jobs.
- Using an English-only embedding model on a French corpus strongly degrades quality.
π Go deeper
- π§ The Journey of a Prompt (Step 2: from token to vector)
- π§© RAG & Agents (how search embeddings feed RAG)
π See also
Splitting text into numeric units (tokens) before an LLM processes it.Core Transformer mechanism that lets each token weight the importance of other tokens in context.Database specialized in storing and searching embedding vectors for RAG.Architecture that combines document retrieval and LLM generation to ground answers in internal sources.