RAG
π Short definition
Method that retrieves relevant passages from a document base, then generates a response with an LLM enriched by that context1.
π Detailed definition
RAG (Retrieval-Augmented Generation) separates two phases:
- Retrieval β query β embeddings β similarity search in a vector database (or hybrid BM25 + vectors).
- Generation β concatenation of system prompt + retrieved chunks + user question β model response.
Variants covered in the vault: Agentic RAG (retrieve/grade/rewrite loops), GraphRAG, Memory Tree for long contexts.
π‘ Why it matters for on-prem AI
- Exploits internal knowledge without retraining the model β suited to GDPR and sovereignty constraints.
- Reduces hallucinations if retrieval is good; does not eliminate them (see pitfalls).
- Coupled with multi-tenant, requires strict index isolation (OWASP LLM08).
β οΈ Common pitfalls
- Naive chunking: pieces too large or cut mid-sentence β useless retrieval.
- Confusing RAG with security: a poisoned document can trigger indirect prompt injection.
- Ignoring evaluation: measure retrieval + faithfulness (RAGAS, golden dataset) β see evaluating a local model.
- Oversizing the model to compensate for a poor index: optimize embeddings and re-ranking first.
π To go deeper
- π§© RAG & Agents β full 2026 architecture
- π§ͺ Evaluating a local model
- LangGraph β agentic orchestration
π See also
Footnotes
-
Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020). https://arxiv.org/abs/2005.11401 β©
AI model trained on very large text corpora, most often of the Transformer type; locally, its performance depends as much on memory as on the model itself.Dense numerical representation of a token or document in vector space.Database specialized in storing and searching embedding vectors for RAG.RAG evolution based on a knowledge graph rather than a vector database alone.Memory architecture organizing documents and summaries in a hierarchy to limit context injected into the LLM.AI SaaS architecture where a single infrastructure serves multiple isolated organizations, with inter-tenant leakage risk in RAG.Attack where untrusted content in the LLM context hijacks system instructions to exfiltrate data or execute unauthorized actions.Framework that evaluates a RAG system by separating retrieval quality from answer faithfulness, to diagnose where the pipeline fails.State-graph framework for orchestrating multi-step LLM agents with loops, memory, and flow control.An LLM's ability to emit structured requests to external functions (API, SQL, code) rather than free-form text.Maximum number of tokens an LLM can handle in active input β drives dynamic inference memory cost.