# HumanEval

> HumanEval is a benchmark of 164 hand-written Python programming problems, each with a function signature, a description, and hidden unit tests, used to measure code generation with the pass@k metric.

- Identifier: PTL-0200
- Category: Benchmarks
- Canonical URL: https://protologue.com/t/humaneval/
- Introduced: 2021

## Description

It established functional correctness, whether generated code passes tests, as the standard for evaluating code models, rather than similarity to a reference solution. It is now saturated and small, so harder and contamination-resistant benchmarks such as LiveCodeBench and SWE-bench have taken its place.

## Broader terms

- [Benchmark](https://protologue.com/t/benchmark/)

## Related terms

- [pass@k](https://protologue.com/t/pass-at-k/)
- [LiveCodeBench](https://protologue.com/t/livecodebench/)
- [SWE-bench](https://protologue.com/t/swe-bench/)

## Sources

- Chen et al. (2021). Evaluating Large Language Models Trained on Code. https://arxiv.org/abs/2107.03374

## Cite this entry

Protologue. (2026). HumanEval. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0200). https://protologue.com/t/humaneval/

License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)