protologue

BIG-Bench Hard

Also called BBH, BIG-bench.

BIG-Bench Hard (BBH) is a set of 23 challenging tasks from the BIG-bench collection on which earlier language models performed below average human raters, used to measure multi-step reasoning.

Description

Its creators showed that chain-of-thought prompting greatly improved results on these tasks, which made BBH a standard test of reasoning-elicitation techniques. BIG-bench itself is a collaborative collection of over 200 tasks.

Sources

  1. Suzgun et al. (2022). Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them.
  2. Srivastava et al. (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.

Cite this entry

Protologue. (2026). BIG-Bench Hard. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0191). https://protologue.com/t/big-bench-hard/

BibTeX
@misc{protologue_big_bench_hard,
  title = {BIG-Bench Hard},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0191},
  url = {https://protologue.com/t/big-bench-hard/}
}

Markdown JSON