# LLM Judge Bias

> LLM judge bias is the set of systematic errors that language models make when grading outputs, notably preferring the answer shown first (position bias), longer answers (verbosity bias), and their own outputs (self-enhancement bias).

- Identifier: PTL-0177
- Category: Failure Modes & Evaluation
- Canonical URL: https://protologue.com/t/llm-judge-bias/
- Also known as: position bias, verbosity bias, self-enhancement bias, self-preference bias
- Introduced: 2023

## Description

These biases can make automated evaluations favor the wrong prompt or model. Common mitigations are evaluating each pair in both orders and counting only consistent verdicts, instructing the judge to ignore length, using clear rubrics and reference answers, and checking judge agreement against a sample of human ratings.

## Related terms

- [LLM-as-a-Judge](https://protologue.com/t/llm-as-a-judge/)
- [Evals](https://protologue.com/t/evals/)
- [Sycophancy](https://protologue.com/t/sycophancy/)

## Sources

- Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. https://arxiv.org/abs/2306.05685

## Cite this entry

Protologue. (2026). LLM Judge Bias. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0177). https://protologue.com/t/llm-judge-bias/

License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)