Skip to content

LLM benchmark

πŸ“ Short definition

Standardized test measuring one or more capabilities of an LLM: reasoning, code, factuality, instruction following, robustness, or performance.

πŸ“– Detailed definition

An LLM benchmark helps compare models quickly, but it always measures a specific protocol. MMLU is not the same as SWE-bench Pro; TruthfulQA is not the same as tokens/s.

Public benchmarks are a starting point, not a final decision.

πŸ’‘ Why it matters for on-prem AI

Locally, the best model is not necessarily the top leaderboard entry. It must also fit VRAM, respect confidentiality, respond fast enough, and pass your domain tests.

⚠️ Common pitfalls

  • Confusing leaderboard score with quality on your documents.
  • Comparing models with different quantization, prompts, or engines.
  • Reading a saturated benchmark: when the best models exceed 95% (SWE-bench Verified in 2026, 97% at the top), the gap between two candidates is noise; move to the harder variant (SWE-bench Pro, Terminal-Bench)1.
  • Comparing two scores from a composite index without checking the index version (Artificial Analysis v4.3 replaced sub-benchmarks on 7 September 2026)2.
  • Confusing the overall ranking with the open-weights ranking: on Arena, no open-weights model is in the top 15 as of Q4 2026 β€” compare open models among themselves3.
  • Ignoring local metrics: TTFT, tokens/s, stability, memory use.

πŸ”— See also

Footnotes

  1. Vals AI, SWE-bench Verified (β€œperformance on this benchmark has saturated, we no longer run this benchmark on new model releases”, top 97.0%), updated 2026-09-01. https://www.vals.ai/benchmarks/swebench ↩

  2. Artificial Analysis, Intelligence Index v4.3 (Terminal-Bench 2.1 β†’ 4.0, AutomationBench-AA, 45% private tasks), 2026-09-07. https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3 ↩

  3. Arena, Text Leaderboard (snapshot of 2026-10-08: no open-weights model in the top 15, kimi-k3-max at rank 16). https://arena.ai/leaderboard/text ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.