protologue

Benchmark Saturation

Also called saturated benchmark.

Benchmark saturation is when the best models score near a benchmark's maximum, so that it no longer distinguishes between them and its remaining errors mostly reflect flawed or ambiguous questions.

Description

Saturation has followed every widely used benchmark, often within a few years, and is a main reason new, harder benchmarks keep appearing. MMLU-Pro and Humanity's Last Exam were both created because earlier benchmarks had saturated. Contamination can make a benchmark look saturated before models have truly mastered it.

Sources

  1. Phan et al. (2025). Humanity's Last Exam.
  2. Wang et al. (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark.

Cite this entry

Protologue. (2026). Benchmark Saturation. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0186). https://protologue.com/t/benchmark-saturation/

BibTeX
@misc{protologue_benchmark_saturation,
  title = {Benchmark Saturation},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0186},
  url = {https://protologue.com/t/benchmark-saturation/}
}

Markdown JSON