protologue

HumanEval

HumanEval is a benchmark of 164 hand-written Python programming problems, each with a function signature, a description, and hidden unit tests, used to measure code generation with the pass@k metric.

Description

It established functional correctness, whether generated code passes tests, as the standard for evaluating code models, rather than similarity to a reference solution. It is now saturated and small, so harder and contamination-resistant benchmarks such as LiveCodeBench and SWE-bench have taken its place.

Sources

  1. Chen et al. (2021). Evaluating Large Language Models Trained on Code.

Cite this entry

Protologue. (2026). HumanEval. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0200). https://protologue.com/t/humaneval/

BibTeX
@misc{protologue_humaneval,
  title = {HumanEval},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0200},
  url = {https://protologue.com/t/humaneval/}
}

Markdown JSON