protologue

Benchmark

Also called LLM benchmark, evaluation benchmark.

A benchmark is a fixed, public set of test tasks with a standard scoring method, used to measure and compare the capabilities of language models.

Description

Reported scores depend on the prompting setup, such as the number of examples, whether chain-of-thought or extended thinking is used, and how many attempts are sampled, so scores are only comparable under the same conditions. Public benchmarks also risk data contamination and eventually saturate. They are useful for choosing models, but an application's own evals are the better guide to how a model will perform on its tasks.

Sources

  1. Liang et al. (2022). Holistic Evaluation of Language Models.
  2. Srivastava et al. (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.

Cite this entry

Protologue. (2026). Benchmark. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0185). https://protologue.com/t/benchmark/

BibTeX
@misc{protologue_benchmark,
  title = {Benchmark},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0185},
  url = {https://protologue.com/t/benchmark/}
}

Markdown JSON