# SimpleQA

> SimpleQA is a benchmark of short, fact-seeking questions with a single indisputable answer, collected adversarially against strong models, that measures whether a model answers factual questions correctly and declines when it does not know.

- Identifier: PTL-0193
- Category: Benchmarks
- Canonical URL: https://protologue.com/t/simpleqa/
- Introduced: 2024

## Description

Answers are graded as correct, incorrect, or not attempted, so it rewards knowing what you don't know as well as knowing facts. It is used to measure hallucination in closed-book settings and the effect of adding search.

## Broader terms

- [Benchmark](https://protologue.com/t/benchmark/)

## Related terms

- [Hallucination](https://protologue.com/t/hallucination/)
- [Confidence Calibration](https://protologue.com/t/confidence-calibration/)
- [Web Search Tool](https://protologue.com/t/web-search-tool/)
- [TruthfulQA](https://protologue.com/t/truthfulqa/)

## Sources

- Wei et al. (2024). Measuring short-form factuality in large language models. https://arxiv.org/abs/2411.04368

## Cite this entry

Protologue. (2026). SimpleQA. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0193). https://protologue.com/t/simpleqa/

License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)