protologue

SWE-bench

Also called SWE-bench Verified.

SWE-bench is a benchmark of real GitHub issues from popular Python repositories, in which a model or agent must edit the codebase so that tests written for the actual fix pass.

Description

It measures practical software engineering in large, unfamiliar codebases rather than isolated functions, and is the standard benchmark for coding agents. Because some original tasks were underspecified or had faulty tests, a 500-task subset screened by human developers, SWE-bench Verified, is the version most often reported.

Sources

  1. Jimenez et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?.
  2. OpenAI (2024). Introducing SWE-bench Verified.

Cite this entry

Protologue. (2026). SWE-bench. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0202). https://protologue.com/t/swe-bench/

BibTeX
@misc{protologue_swe_bench,
  title = {SWE-bench},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0202},
  url = {https://protologue.com/t/swe-bench/}
}

Markdown JSON