Vector database
π Short definition
Database that stores numeric representations (vectors) of documents and finds those closest to a query by mathematical similarity.
π Detailed definition
Each document is converted to a vector (embedding) by a specialized model. On search, the query is also embedded and the database returns documents whose vectors are closest (cosine similarity or dot product).
Common solutions in 2026: Qdrant, Milvus, Chroma (lightweight local), Weaviate. Most are open source and deployable on-prem.
Main limit in classic RAG: search is βblindββit finds textually close passages but can miss complex semantic links or entity relationships. That is what GraphRAG tries to address.
π‘ Why it matters for on-prem AI
Central building block of standard RAG. Must be deployed locally to preserve data sovereignty.
β οΈ Common pitfalls
- Returning too many hits (e.g. top-20) bloats the prompt and saturates the model context window.
- Embedding quality is critical: a poor embedding model gives poor search results regardless of the database.
- Cross-tenant data leakage β in a multi-user environment, a query can return documents from another tenant if no isolation filter (namespace,
tenant_idin metadata) is applied on every search. ACLs at ingestion are not enough: the filter must be re-evaluated at query time. See LLM08 β Vector & Embedding Weaknesses.
π Go deeper
- π§© RAG & Agents (full RAG pipeline and agentic alternatives)
π See also
RAG evolution based on a knowledge graph rather than a vector database alone.Maximum number of tokens an LLM can handle in active input β drives dynamic inference memory cost.Architecture that combines document retrieval and LLM generation to ground answers in internal sources.