protologue

MMLU-Pro

MMLU-Pro is a harder, reasoning-focused successor to MMLU that offers ten answer options instead of four and replaces trivial or noisy questions with more challenging ones.

Description

More options lower the score obtainable by guessing, and the harder questions make chain-of-thought prompting matter more than it does on MMLU. Its creators also reported that scores vary less with the wording of the prompt than on the original benchmark.

Sources

  1. Wang et al. (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark.

Cite this entry

Protologue. (2026). MMLU-Pro. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0188). https://protologue.com/t/mmlu-pro/

BibTeX
@misc{protologue_mmlu_pro,
  title = {MMLU-Pro},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0188},
  url = {https://protologue.com/t/mmlu-pro/}
}

Markdown JSON