# Benchmark

> A benchmark is a fixed, public set of test tasks with a standard scoring method, used to measure and compare the capabilities of language models.

- Identifier: PTL-0185
- Category: Benchmarks
- Canonical URL: https://protologue.com/t/benchmark/
- Also known as: LLM benchmark, evaluation benchmark

## Description

Reported scores depend on the prompting setup, such as the number of examples, whether chain-of-thought or extended thinking is used, and how many attempts are sampled, so scores are only comparable under the same conditions. Public benchmarks also risk data contamination and eventually saturate. They are useful for choosing models, but an application's own evals are the better guide to how a model will perform on its tasks.

## Narrower terms

- [Benchmark Saturation](https://protologue.com/t/benchmark-saturation/)
- [MMLU](https://protologue.com/t/mmlu/)
- [MMLU-Pro](https://protologue.com/t/mmlu-pro/)
- [GPQA](https://protologue.com/t/gpqa/)
- [Humanity's Last Exam](https://protologue.com/t/humanitys-last-exam/)
- [BIG-Bench Hard](https://protologue.com/t/big-bench-hard/)
- [ARC-AGI](https://protologue.com/t/arc-agi/)
- [SimpleQA](https://protologue.com/t/simpleqa/)
- [TruthfulQA](https://protologue.com/t/truthfulqa/)
- [HellaSwag](https://protologue.com/t/hellaswag/)
- [MMMU](https://protologue.com/t/mmmu/)
- [GSM8K](https://protologue.com/t/gsm8k/)
- [MATH](https://protologue.com/t/math-benchmark/)
- [AIME](https://protologue.com/t/aime/)
- [HumanEval](https://protologue.com/t/humaneval/)
- [LiveCodeBench](https://protologue.com/t/livecodebench/)
- [SWE-bench](https://protologue.com/t/swe-bench/)
- [Terminal-Bench](https://protologue.com/t/terminal-bench/)
- [τ-bench](https://protologue.com/t/tau-bench/)
- [OSWorld](https://protologue.com/t/osworld/)
- [GAIA](https://protologue.com/t/gaia/)
- [BrowseComp](https://protologue.com/t/browsecomp/)
- [IFEval](https://protologue.com/t/ifeval/)
- [MT-Bench](https://protologue.com/t/mt-bench/)
- [Chatbot Arena](https://protologue.com/t/chatbot-arena/)
- [RULER](https://protologue.com/t/ruler/)

## Related terms

- [Evals](https://protologue.com/t/evals/)
- [Data Contamination](https://protologue.com/t/data-contamination/)
- [pass@k](https://protologue.com/t/pass-at-k/)

## Sources

- Liang et al. (2022). Holistic Evaluation of Language Models. https://arxiv.org/abs/2211.09110
- Srivastava et al. (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. https://arxiv.org/abs/2206.04615

## Cite this entry

Protologue. (2026). Benchmark. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0185). https://protologue.com/t/benchmark/

License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)