Skip to content

🧪 Evaluate a local model

Choosing a local model is not about picking the first name at the top of a leaderboard. A model can excel at math, be mediocre at domain-specific French, fast but hallucinated, or very good at RAG but risky for editing a repository.


Three levels of evaluation

1. Public benchmarks

Public benchmarks give a first orientation, but their predictive value for enterprise use is severely limited in 2026.

Benchmarks remain useful for roughly sorting model families, or for checking very targeted capabilities (formal reasoning, syntactically correct code). For that, prefer domain-specific tests and evaluations on real data (SWE-bench for code, for example, because it measures on real GitHub issues, not memorizable exercises).

FamilyExamplesReal utilityLimits
General knowledgeMMLU, MMLU-Pro, GPQA Diamondrough sort between familiesMMLU saturated and contaminated; MMLU-Pro and GPQA still discriminate (Q4 2026) but do not predict domain work
Math / reasoningGSM8K, MATHcheck formal logicnot representative of prose tasks
Instruction followingIFEval, MT-Benchconversational qualityresults vary by language
FactualityTruthfulQA, FActScore, HaluEvalresistance to false beliefsmeasures general factuality, not your domain
CodeHumanEval, MBPP, SWE-bench Pro, Terminal-Benchgeneration/editing abilitySWE-bench Verified has been saturated since 2026; prefer SWE-bench Pro V21
Holistic evaluationHELMmulti-metric profileuseful as complement, not replacement for domain tests

HELM reminds us that a model is not simply “good” or “bad”: it has a profile — accuracy, robustness, calibration, bias, toxicity, efficiency2. But even a favorable HELM profile guarantees nothing on your data.

Composite indices change methodology without renaming the site: Artificial Analysis’s Intelligence Index moved to v4.3 on 7 September 2026 (Terminal-Bench 2.1 → 4.0, τ³-Banking replaced by AutomationBench-AA, 45% private tasks); two scores taken before and after are not comparable. Always note the index version next to the score3.

2. Your domain test bench

The real comparison starts with a golden dataset: a small set of representative questions, validated by a competent human.

A good test set contains:

  • 20 to 50 simple questions where the expected answer is clear;
  • 20 to 50 hard, ambiguous, or trap questions;
  • 10 to 20 “do not answer” cases: missing information, out-of-scope request, conflict between sources;
  • a few long questions that test context window and KV Cache;
  • mandatory output formats: JSON, table, short summary, source citation.

For each question, store:

FieldExample
question“What is the procedure for validating an agent PR?”
expected_sourcedocument path or reference excerpt
expected_answershort answer or grading criteria
risklow, medium, critical
typeRAG, reasoning, summary, extraction, refusal

3. Human validation

Automatic scores speed up triage, but the final decision must remain human for critical uses.


Which KPIs to measure?

Response quality

KPIQuestion
AccuracyIs the answer correct?
CompletenessDoes it cover essential elements?
CoherenceDoes it contradict itself between paragraphs or turns?
Instruction adherenceDoes it follow the requested format?
Appropriate refusalDoes it say “I don’t know” when the source is missing?

Hallucinations and factuality

TruthfulQA tests a model’s ability to avoid false but plausible answers, often learned by imitating human text4. FActScore goes further on long text: it splits the answer into atomic facts and checks what proportion is supported by a reliable source5.

For an internal guide, the most useful KPI is often:

Critical hallucination rate=dangerously false answerstotal answers\text{Critical hallucination rate} = \frac{\text{dangerously false answers}}{\text{total answers}}

A critical hallucination is not just an error: it is an answer that could trigger a bad business decision.

RAG

For a RAG architecture, split the problem in two:

ComponentKPI
Retrievalcontext precision, context recall, correct source rate in top-k
Generationfaithfulness, answer relevancy, correct citation

RAGAS proposes evaluating faithfulness of the answer to context, answer relevance, and quality of retrieved context, without always requiring a human reference answer6.

Code and agents

For a custodian agent or a command-line coding tool (Aider — no commits since May 2026 —, OpenHands or an ACP harness), completion benchmarks are not enough. Test real editing:

  • does the patch compile?
  • do tests pass?
  • is the diff minimal?
  • does the model respect allowed files?
  • does it break Markdown, YAML frontmatter, or wikilinks?
  • does it loop on the same fix?

SWE-bench measures this capability from real GitHub issues: the model must produce a patch and the repo tests serve as arbiter7. Use the Pro V2 variant (Scale, September 2026): the Verified variant is saturated (97% at the top), not merely contaminated, and is no longer run by independent evaluators1. That is much closer to a maintenance agent than a simple function-generation benchmark.

Local performance

Even if this chapter focuses mainly on quality, always measure:

  • TTFT;
  • tokens/s;
  • VRAM used at idle and under load;
  • KV Cache consumption;
  • throughput with 1, 5, 20 simultaneous users;
  • stability after 1 hour of load.

Use an AI as judge?

Yes, but carefully.

The judge model (LLM-as-a-judge) is useful to pre-sort many open-ended responses. Work around MT-Bench and Chatbot Arena shows that a strong judge can approach human agreement on conversational preferences, but with documented biases: answer position, verbosity, preference for its own model family8.

Good practices:

  1. Use an explicit rubric: accuracy, source, format, concision, risk.
  2. Ask for a short justification, not only a score.
  3. Compare in double-blind: anonymized models A/B.
  4. Reverse answer order to detect position bias.
  5. Do not use the same model as candidate and judge.
  6. Manually review a sample of decisions.

Concrete 7-step protocol

Step 1 — Define the task

Examples:

  • RAG assistant for HR documents;
  • custodian agent that fixes a Markdown vault;
  • legal summary;
  • JSON extraction from invoices;
  • level-1 internal support.

Step 2 — Define acceptance thresholds

Example for a document assistant:

KPIThreshold
Answer with correct source≥ 95% on simple questions
Critical hallucination0 tolerated
Correct refusal when source missing≥ 90%
TTFT< 2 s
Throughput≥ 10 tokens/s per interactive user

Step 3 — Build the golden dataset

Start small. A CSV or JSONL file is enough:

{"id":"rag-001","question":"Which scenario fits 10 users on a 70B model?","expected_source":"04-blueprints/scenario-b-sme-appliance.md","risk":"medium","type":"rag"}

Step 4 — Fix parameters

To compare fairly:

  • same system prompt;
  • same temperature;
  • same context size;
  • same quantization;
  • same inference engine;
  • same hardware;
  • same model version.

Step 5 — Run multiple passes

A single pass is not enough. LLMs are non-deterministic whenever temperature is above zero.

For critical tasks, run at least 3 passes and note:

  • average score;
  • worst answer;
  • variability between runs;
  • recurring errors.

Step 6 — Analyze errors

Classify each error:

CategoryExample
Retrieval missThe right document was not retrieved
HallucinationThe model invents a non-existent policy
Bad formatInvalid JSON, broken table
OverconfidenceAnswers when the source is missing
Faulty reasoningGood source, wrong conclusion
RegressionOld model answered correctly, new one fails

Step 7 — Decide

The best model is rarely the largest. The right model is the one that meets thresholds, fits your VRAM, respects confidentiality, and stays operable.


Decision matrix

NeedPriority metricUseful public benchmarkEssential local test
General chathuman preference, instruction followingMT-Bench, Arena (formerly Chatbot Arena), IFEval/IFBenchanonymized domain conversations
Document RAGfaithfulness, context recallRAGASsourced questions on your documents
Code agentcorrect patch, tests passSWE-bench Pro V2, Terminal-Benchsimulated PRs on your repo
Legal / medical summaryfactuality, critical omissionsFActScore, TruthfulQAexpert human review
SMB deploymentTTFT, tokens/s, stabilityengine benchmarksconcurrent load on target hardware

See also

Progressive deployment strategy

Beyond out-of-production evaluation, progressive rollout reduces the risk of exposing users to an uncontrolled model. Three sequential phases constitute the recommended pattern.

Phase 1 — Mock-First

Before connecting a real LLM, all async infrastructure (API, queue, workers, frontend reactions) is validated with deterministic mock responses. This phase verifies that the system handles latency correctly and that the interface degrades gracefully — without introducing the non-determinism of a real model.

Phase 2 — Shadow Mode

The real LLM is connected to production traffic, but its outputs are only logged — never shown to users. This phase measures:

  • JSON formatting error rate (does the model reliably respect the output schema?);
  • p95 latency under real load;
  • RAG relevance (are retrieved documents useful for the query?).

Typical duration: 1 to 2 weeks on real traffic before moving to the next phase.

Phase 3 — Human-in-the-Loop activation

AI results become visible to users. Any write action (auto-classification, pre-fill, content modification) requires explicit human confirmation before execution. This is the last safety barrier before full automation.

flowchart LR
    A[Mock-First] --> B[Shadow Mode]
    B --> C{Metrics OK?}
    C -- No --> B
    C -- Yes --> D[HITL activation]
    D --> E{Confidence ≥ threshold?}
    E -- No --> F[Silent degradation]
    E -- Yes --> G[Automatic action]

Sources

Footnotes

  1. Vals AI, SWE-bench Verified (“performance on this benchmark has saturated, we no longer run this benchmark on new model releases”, top 97.0%), updated 2026-09-01. https://www.vals.ai/benchmarks/swebench; Scale AI, SWE-Bench Pro V2 (642 tasks, 11 repositories), 2026-09-22. https://labs.scale.com/leaderboard/swe_bench_pro_public_v2; Aider-AI, aider repository (last commit on 2026-05-22), accessed 2026-10-10. https://github.com/Aider-AI/aider ↩ ↩2

  2. Stanford CRFM, Holistic Evaluation of Language Models (HELM). https://crfm.stanford.edu/helm/ ↩

  3. Artificial Analysis, Intelligence Index v4.3 (Terminal-Bench 4.0, AutomationBench-AA, 45% private tasks), 2026-09-07. https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3 ↩

  4. Lin, Hilton, Evans, TruthfulQA: Measuring How Models Mimic Human Falsehoods, 2021. https://arxiv.org/abs/2109.07958 ↩

  5. Min et al., FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, EMNLP 2023. https://aclanthology.org/2023.emnlp-main.741/ ↩

  6. Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation, EACL 2024. https://aclanthology.org/2024.eacl-demo.16/ ↩

  7. Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, ICLR 2024. https://www.swebench.com/original.html ↩

  8. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023. https://arxiv.org/abs/2306.05685 ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.