LLM-as-a-judge
π Short definition
Method using an LLM to score, compare, or explain the quality of answers from other models.
π Detailed definition
A judge LLM can:
- compare two answers to the same question;
- assign a score against a rubric;
- check whether an answer follows instructions;
- flag likely inconsistencies or hallucinations.
Useful to speed up open-ended evaluation, but imperfect: judge LLMs may favor long answers, be sensitive to answer order, or prefer their own style.
π‘ Why it matters for on-prem AI
When choosing a local model, a judge LLM can pre-sort many answers before human review. It should not alone decide critical use cases.
β οΈ Common pitfalls
- Using the same model as candidate and judge.
- Forgetting to swap A/B answer order in comparisons.
- Scoring without an explicit rubric.
- Confusing automatic judgment with business validation.
π See also
AI model trained on very large text corpora, most often of the Transformer type; locally, its performance depends as much on memory as on the model itself.Standardized test suite to compare capabilities, limits, and risks of language models.Framework that evaluates a RAG system by separating retrieval quality from answer faithfulness, to diagnose where the pipeline fails.