LLM benchmark
π Short definition
Standardized test measuring one or more capabilities of an LLM: reasoning, code, factuality, instruction following, robustness, or performance.
π Detailed definition
An LLM benchmark helps compare models quickly, but it always measures a specific protocol. MMLU is not the same as SWE-bench Pro; TruthfulQA is not the same as tokens/s.
Public benchmarks are a starting point, not a final decision.
π‘ Why it matters for on-prem AI
Locally, the best model is not necessarily the top leaderboard entry. It must also fit VRAM, respect confidentiality, respond fast enough, and pass your domain tests.
β οΈ Common pitfalls
- Confusing leaderboard score with quality on your documents.
- Comparing models with different quantization, prompts, or engines.
- Reading a saturated benchmark: when the best models exceed 95% (SWE-bench Verified in 2026, 97% at the top), the gap between two candidates is noise; move to the harder variant (SWE-bench Pro, Terminal-Bench)1.
- Comparing two scores from a composite index without checking the index version (Artificial Analysis v4.3 replaced sub-benchmarks on 7 September 2026)2.
- Confusing the overall ranking with the open-weights ranking: on Arena, no open-weights model is in the top 15 as of Q4 2026 β compare open models among themselves3.
- Ignoring local metrics: TTFT, tokens/s, stability, memory use.
π See also
- Evaluate a local model
- LLM-as-a-judge
- RAGAS
- Choose your local model β read a leaderboard without getting it wrong
Footnotes
-
Vals AI, SWE-bench Verified (βperformance on this benchmark has saturated, we no longer run this benchmark on new model releasesβ, top 97.0%), updated 2026-09-01. https://www.vals.ai/benchmarks/swebench β©
-
Artificial Analysis, Intelligence Index v4.3 (Terminal-Bench 2.1 β 4.0, AutomationBench-AA, 45% private tasks), 2026-09-07. https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3 β©
-
Arena, Text Leaderboard (snapshot of 2026-10-08: no open-weights model in the top 15, kimi-k3-max at rank 16). https://arena.ai/leaderboard/text β©