protologue

Benchmarks

Standard test sets used to measure and compare language models, what each one measures, how it is scored, the prompting setup scores are reported under, and its known limitations.

Definitions

AIME
AIME, the American Invitational Mathematics Examination, is a US high-school mathematics competition whose problems, each with an integer answer, are widely used as a benchmark for reasoning models, usually identified by year, as in AIME 2024 or AIME 2025.
ARC-AGI
ARC-AGI (the Abstraction and Reasoning Corpus) is a benchmark of visual grid puzzles in which a model must infer a transformation rule from a few examples and apply it to a new grid, designed to measure how efficiently a system acquires new skills.
Benchmark
A benchmark is a fixed, public set of test tasks with a standard scoring method, used to measure and compare the capabilities of language models.
Benchmark Saturation
Benchmark saturation is when the best models score near a benchmark's maximum, so that it no longer distinguishes between them and its remaining errors mostly reflect flawed or ambiguous questions.
BIG-Bench Hard
BIG-Bench Hard (BBH) is a set of 23 challenging tasks from the BIG-bench collection on which earlier language models performed below average human raters, used to measure multi-step reasoning.
BrowseComp
BrowseComp is a benchmark of hard-to-find but easy-to-verify questions that require an agent to search the web persistently and creatively, often across many pages, to locate an entangled piece of information.
Chatbot Arena
Chatbot Arena is a public platform where people chat with two anonymous models side by side and vote for the better response, with the votes aggregated into a leaderboard of model ratings.
GAIA
GAIA is a benchmark for general AI assistants made of questions that are conceptually simple for people but require several steps of web browsing, file reading, multimodal understanding, and tool use to answer.
GPQA
GPQA (Graduate-Level Google-Proof Q&A) is a benchmark of difficult multiple-choice questions in biology, physics, and chemistry, written by domain experts so that skilled non-experts cannot answer them reliably even with unrestricted web access.
GSM8K
GSM8K is a benchmark of about 8,500 grade-school math word problems that each take several steps of basic arithmetic to solve, used to measure multi-step mathematical reasoning.
HellaSwag
HellaSwag is a commonsense reasoning benchmark in which a model must pick the most plausible continuation of a description of an everyday situation from four options, with wrong options generated and filtered to fool models.
HumanEval
HumanEval is a benchmark of 164 hand-written Python programming problems, each with a function signature, a description, and hidden unit tests, used to measure code generation with the pass@k metric.
Humanity's Last Exam
Humanity's Last Exam (HLE) is a benchmark of 2,500 expert-written questions across dozens of academic subjects, some with images, designed to remain hard for frontier models after earlier academic benchmarks saturated.
IFEval
IFEval (Instruction-Following Eval) is a benchmark of prompts containing instructions whose compliance can be checked automatically, such as "write at least 300 words" or "do not use any commas", used to measure instruction following without a human or model judge.
LiveCodeBench
LiveCodeBench is a coding benchmark that continuously collects new problems from programming contests, such as LeetCode, AtCoder, and Codeforces, so that models can be evaluated only on problems published after their training data was collected.
MATH
MATH is a benchmark of 12,500 competition mathematics problems, spanning seven subjects and five difficulty levels, each with a full step-by-step solution.
MMLU
MMLU (Massive Multitask Language Understanding) is a benchmark of four-option multiple-choice questions across 57 subjects, from elementary mathematics to law and medicine, that measures a model's breadth of knowledge and problem solving.
MMLU-Pro
MMLU-Pro is a harder, reasoning-focused successor to MMLU that offers ten answer options instead of four and replaces trivial or noisy questions with more challenging ones.
MMMU
MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark of college-level questions that require reasoning over images, such as charts, diagrams, tables, and chemical structures, together with text, across 30 subjects in six disciplines.
MT-Bench
MT-Bench is a benchmark of 80 two-turn questions across categories such as writing, reasoning, math, and coding, in which a strong LLM acts as the judge and scores each response.
OSWorld
OSWorld is a benchmark of open-ended tasks that computer-use agents must complete in real operating systems and applications, graded by scripts that check the resulting state of the computer.
RULER
RULER is a synthetic long-context benchmark that extends needle-in-a-haystack retrieval with harder tasks, such as multiple needles, multi-hop tracing, and aggregation, at configurable context lengths, to measure how much of its context window a model can actually use.
SimpleQA
SimpleQA is a benchmark of short, fact-seeking questions with a single indisputable answer, collected adversarially against strong models, that measures whether a model answers factual questions correctly and declines when it does not know.
SWE-bench
SWE-bench is a benchmark of real GitHub issues from popular Python repositories, in which a model or agent must edit the codebase so that tests written for the actual fix pass.
Terminal-Bench
Terminal-Bench is a benchmark of hard, realistic tasks that an AI agent must complete by running commands in a containerized Linux terminal, graded by automated tests of the final state of the environment.
TruthfulQA
TruthfulQA is a benchmark of questions crafted so that some people would answer them falsely because of common misconceptions, used to measure whether a model repeats popular falsehoods instead of answering truthfully.
τ-bench
τ-bench is a benchmark in which an agent with API tools and a written domain policy, for example for airline or retail customer service, converses with a simulated user and is judged by whether the final database state is correct.