Skip to content

LLM-as-a-judge

πŸ“ Short definition

Method using an LLM to score, compare, or explain the quality of answers from other models.

πŸ“– Detailed definition

A judge LLM can:

  • compare two answers to the same question;
  • assign a score against a rubric;
  • check whether an answer follows instructions;
  • flag likely inconsistencies or hallucinations.

Useful to speed up open-ended evaluation, but imperfect: judge LLMs may favor long answers, be sensitive to answer order, or prefer their own style.

πŸ’‘ Why it matters for on-prem AI

When choosing a local model, a judge LLM can pre-sort many answers before human review. It should not alone decide critical use cases.

⚠️ Common pitfalls

  • Using the same model as candidate and judge.
  • Forgetting to swap A/B answer order in comparisons.
  • Scoring without an explicit rubric.
  • Confusing automatic judgment with business validation.

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.