protologue

SimpleQA

SimpleQA is a benchmark of short, fact-seeking questions with a single indisputable answer, collected adversarially against strong models, that measures whether a model answers factual questions correctly and declines when it does not know.

Description

Answers are graded as correct, incorrect, or not attempted, so it rewards knowing what you don't know as well as knowing facts. It is used to measure hallucination in closed-book settings and the effect of adding search.

Sources

  1. Wei et al. (2024). Measuring short-form factuality in large language models.

Cite this entry

Protologue. (2026). SimpleQA. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0193). https://protologue.com/t/simpleqa/

BibTeX
@misc{protologue_simpleqa,
  title = {SimpleQA},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0193},
  url = {https://protologue.com/t/simpleqa/}
}

Markdown JSON