protologue

LLM Judge Bias

Also called position bias, verbosity bias, self-enhancement bias, self-preference bias.

LLM judge bias is the set of systematic errors that language models make when grading outputs, notably preferring the answer shown first (position bias), longer answers (verbosity bias), and their own outputs (self-enhancement bias).

Description

These biases can make automated evaluations favor the wrong prompt or model. Common mitigations are evaluating each pair in both orders and counting only consistent verdicts, instructing the judge to ignore length, using clear rubrics and reference answers, and checking judge agreement against a sample of human ratings.

Sources

  1. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.

Cite this entry

Protologue. (2026). LLM Judge Bias. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0177). https://protologue.com/t/llm-judge-bias/

BibTeX
@misc{protologue_llm_judge_bias,
  title = {LLM Judge Bias},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0177},
  url = {https://protologue.com/t/llm-judge-bias/}
}

Markdown JSON