# BIG-Bench Hard

> BIG-Bench Hard (BBH) is a set of 23 challenging tasks from the BIG-bench collection on which earlier language models performed below average human raters, used to measure multi-step reasoning.

- Identifier: PTL-0191
- Category: Benchmarks
- Canonical URL: https://protologue.com/t/big-bench-hard/
- Also known as: BBH, BIG-bench
- Introduced: 2022

## Description

Its creators showed that chain-of-thought prompting greatly improved results on these tasks, which made BBH a standard test of reasoning-elicitation techniques. BIG-bench itself is a collaborative collection of over 200 tasks.

## Broader terms

- [Benchmark](https://protologue.com/t/benchmark/)

## Related terms

- [Chain-of-Thought Prompting](https://protologue.com/t/chain-of-thought/)
- [Few-shot Prompting](https://protologue.com/t/few-shot-prompting/)

## Sources

- Suzgun et al. (2022). Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. https://arxiv.org/abs/2210.09261
- Srivastava et al. (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. https://arxiv.org/abs/2206.04615

## Cite this entry

Protologue. (2026). BIG-Bench Hard. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0191). https://protologue.com/t/big-bench-hard/

License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)