🧪 Evaluate a local model
Choosing a local model is not about picking the first name at the top of a leaderboard. A model can excel at math, be mediocre at domain-specific French, fast but hallucinated, or very good at RAG but risky for editing a repository.
Three levels of evaluation
1. Public benchmarks
Public benchmarks give a first orientation, but their predictive value for enterprise use is severely limited in 2026.
Benchmarks remain useful for roughly sorting model families, or for checking very targeted capabilities (formal reasoning, syntactically correct code). For that, prefer domain-specific tests and evaluations on real data (SWE-bench for code, for example, because it measures on real GitHub issues, not memorizable exercises).
| Family | Examples | Real utility | Limits |
|---|---|---|---|
| General knowledge | MMLU, MMLU-Pro, GPQA Diamond | rough sort between families | MMLU saturated and contaminated; MMLU-Pro and GPQA still discriminate (Q4 2026) but do not predict domain work |
| Math / reasoning | GSM8K, MATH | check formal logic | not representative of prose tasks |
| Instruction following | IFEval, MT-Bench | conversational quality | results vary by language |
| Factuality | TruthfulQA, FActScore, HaluEval | resistance to false beliefs | measures general factuality, not your domain |
| Code | HumanEval, MBPP, SWE-bench Pro, Terminal-Bench | generation/editing ability | SWE-bench Verified has been saturated since 2026; prefer SWE-bench Pro V21 |
| Holistic evaluation | HELM | multi-metric profile | useful as complement, not replacement for domain tests |
HELM reminds us that a model is not simply “good” or “bad”: it has a profile — accuracy, robustness, calibration, bias, toxicity, efficiency2. But even a favorable HELM profile guarantees nothing on your data.
Composite indices change methodology without renaming the site: Artificial Analysis’s Intelligence Index moved to v4.3 on 7 September 2026 (Terminal-Bench 2.1 → 4.0, τ³-Banking replaced by AutomationBench-AA, 45% private tasks); two scores taken before and after are not comparable. Always note the index version next to the score3.
2. Your domain test bench
The real comparison starts with a golden dataset: a small set of representative questions, validated by a competent human.
A good test set contains:
- 20 to 50 simple questions where the expected answer is clear;
- 20 to 50 hard, ambiguous, or trap questions;
- 10 to 20 “do not answer” cases: missing information, out-of-scope request, conflict between sources;
- a few long questions that test context window and KV Cache;
- mandatory output formats: JSON, table, short summary, source citation.
For each question, store:
| Field | Example |
|---|---|
question | “What is the procedure for validating an agent PR?” |
expected_source | document path or reference excerpt |
expected_answer | short answer or grading criteria |
risk | low, medium, critical |
type | RAG, reasoning, summary, extraction, refusal |
3. Human validation
Automatic scores speed up triage, but the final decision must remain human for critical uses.
Which KPIs to measure?
Response quality
| KPI | Question |
|---|---|
| Accuracy | Is the answer correct? |
| Completeness | Does it cover essential elements? |
| Coherence | Does it contradict itself between paragraphs or turns? |
| Instruction adherence | Does it follow the requested format? |
| Appropriate refusal | Does it say “I don’t know” when the source is missing? |
Hallucinations and factuality
TruthfulQA tests a model’s ability to avoid false but plausible answers, often learned by imitating human text4. FActScore goes further on long text: it splits the answer into atomic facts and checks what proportion is supported by a reliable source5.
For an internal guide, the most useful KPI is often:
A critical hallucination is not just an error: it is an answer that could trigger a bad business decision.
RAG
For a RAG architecture, split the problem in two:
| Component | KPI |
|---|---|
| Retrieval | context precision, context recall, correct source rate in top-k |
| Generation | faithfulness, answer relevancy, correct citation |
RAGAS proposes evaluating faithfulness of the answer to context, answer relevance, and quality of retrieved context, without always requiring a human reference answer6.
Code and agents
For a custodian agent or a command-line coding tool (Aider — no commits since May 2026 —, OpenHands or an ACP harness), completion benchmarks are not enough. Test real editing:
- does the patch compile?
- do tests pass?
- is the diff minimal?
- does the model respect allowed files?
- does it break Markdown, YAML frontmatter, or wikilinks?
- does it loop on the same fix?
SWE-bench measures this capability from real GitHub issues: the model must produce a patch and the repo tests serve as arbiter7. Use the Pro V2 variant (Scale, September 2026): the Verified variant is saturated (97% at the top), not merely contaminated, and is no longer run by independent evaluators1. That is much closer to a maintenance agent than a simple function-generation benchmark.
Local performance
Even if this chapter focuses mainly on quality, always measure:
- TTFT;
- tokens/s;
- VRAM used at idle and under load;
- KV Cache consumption;
- throughput with 1, 5, 20 simultaneous users;
- stability after 1 hour of load.
Use an AI as judge?
Yes, but carefully.
The judge model (LLM-as-a-judge) is useful to pre-sort many open-ended responses. Work around MT-Bench and Chatbot Arena shows that a strong judge can approach human agreement on conversational preferences, but with documented biases: answer position, verbosity, preference for its own model family8.
Good practices:
- Use an explicit rubric: accuracy, source, format, concision, risk.
- Ask for a short justification, not only a score.
- Compare in double-blind: anonymized models A/B.
- Reverse answer order to detect position bias.
- Do not use the same model as candidate and judge.
- Manually review a sample of decisions.
Concrete 7-step protocol
Step 1 — Define the task
Examples:
- RAG assistant for HR documents;
- custodian agent that fixes a Markdown vault;
- legal summary;
- JSON extraction from invoices;
- level-1 internal support.
Step 2 — Define acceptance thresholds
Example for a document assistant:
| KPI | Threshold |
|---|---|
| Answer with correct source | ≥ 95% on simple questions |
| Critical hallucination | 0 tolerated |
| Correct refusal when source missing | ≥ 90% |
| TTFT | < 2 s |
| Throughput | ≥ 10 tokens/s per interactive user |
Step 3 — Build the golden dataset
Start small. A CSV or JSONL file is enough:
{"id":"rag-001","question":"Which scenario fits 10 users on a 70B model?","expected_source":"04-blueprints/scenario-b-sme-appliance.md","risk":"medium","type":"rag"}Step 4 — Fix parameters
To compare fairly:
- same system prompt;
- same temperature;
- same context size;
- same quantization;
- same inference engine;
- same hardware;
- same model version.
Step 5 — Run multiple passes
A single pass is not enough. LLMs are non-deterministic whenever temperature is above zero.
For critical tasks, run at least 3 passes and note:
- average score;
- worst answer;
- variability between runs;
- recurring errors.
Step 6 — Analyze errors
Classify each error:
| Category | Example |
|---|---|
| Retrieval miss | The right document was not retrieved |
| Hallucination | The model invents a non-existent policy |
| Bad format | Invalid JSON, broken table |
| Overconfidence | Answers when the source is missing |
| Faulty reasoning | Good source, wrong conclusion |
| Regression | Old model answered correctly, new one fails |
Step 7 — Decide
The best model is rarely the largest. The right model is the one that meets thresholds, fits your VRAM, respects confidentiality, and stays operable.
Decision matrix
| Need | Priority metric | Useful public benchmark | Essential local test |
|---|---|---|---|
| General chat | human preference, instruction following | MT-Bench, Arena (formerly Chatbot Arena), IFEval/IFBench | anonymized domain conversations |
| Document RAG | faithfulness, context recall | RAGAS | sourced questions on your documents |
| Code agent | correct patch, tests pass | SWE-bench Pro V2, Terminal-Bench | simulated PRs on your repo |
| Legal / medical summary | factuality, critical omissions | FActScore, TruthfulQA | expert human review |
| SMB deployment | TTFT, tokens/s, stability | engine benchmarks | concurrent load on target hardware |
See also
Progressive deployment strategy
Beyond out-of-production evaluation, progressive rollout reduces the risk of exposing users to an uncontrolled model. Three sequential phases constitute the recommended pattern.
Phase 1 — Mock-First
Before connecting a real LLM, all async infrastructure (API, queue, workers, frontend reactions) is validated with deterministic mock responses. This phase verifies that the system handles latency correctly and that the interface degrades gracefully — without introducing the non-determinism of a real model.
Phase 2 — Shadow Mode
The real LLM is connected to production traffic, but its outputs are only logged — never shown to users. This phase measures:
- JSON formatting error rate (does the model reliably respect the output schema?);
- p95 latency under real load;
- RAG relevance (are retrieved documents useful for the query?).
Typical duration: 1 to 2 weeks on real traffic before moving to the next phase.
Phase 3 — Human-in-the-Loop activation
AI results become visible to users. Any write action (auto-classification, pre-fill, content modification) requires explicit human confirmation before execution. This is the last safety barrier before full automation.
flowchart LR
A[Mock-First] --> B[Shadow Mode]
B --> C{Metrics OK?}
C -- No --> B
C -- Yes --> D[HITL activation]
D --> E{Confidence ≥ threshold?}
E -- No --> F[Silent degradation]
E -- Yes --> G[Automatic action]
Sources
Footnotes
-
Vals AI, SWE-bench Verified (“performance on this benchmark has saturated, we no longer run this benchmark on new model releases”, top 97.0%), updated 2026-09-01. https://www.vals.ai/benchmarks/swebench; Scale AI, SWE-Bench Pro V2 (642 tasks, 11 repositories), 2026-09-22. https://labs.scale.com/leaderboard/swe_bench_pro_public_v2; Aider-AI,
aiderrepository (last commit on 2026-05-22), accessed 2026-10-10. https://github.com/Aider-AI/aider ↩ ↩2 -
Stanford CRFM, Holistic Evaluation of Language Models (HELM). https://crfm.stanford.edu/helm/ ↩
-
Artificial Analysis, Intelligence Index v4.3 (Terminal-Bench 4.0, AutomationBench-AA, 45% private tasks), 2026-09-07. https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3 ↩
-
Lin, Hilton, Evans, TruthfulQA: Measuring How Models Mimic Human Falsehoods, 2021. https://arxiv.org/abs/2109.07958 ↩
-
Min et al., FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, EMNLP 2023. https://aclanthology.org/2023.emnlp-main.741/ ↩
-
Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation, EACL 2024. https://aclanthology.org/2024.eacl-demo.16/ ↩
-
Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, ICLR 2024. https://www.swebench.com/original.html ↩
-
Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023. https://arxiv.org/abs/2306.05685 ↩