# pass@k

> pass@k is an evaluation metric for code generation, and other tasks with automatic checks, defined as the probability that at least one of k sampled solutions to a problem passes its tests.

- Identifier: PTL-0180
- Category: Failure Modes & Evaluation
- Canonical URL: https://protologue.com/t/pass-at-k/
- Also known as: pass@1, pass at k
- Introduced: 2021

## Description

It was popularized with an unbiased estimator computed from n ≥ k samples per problem. pass@1 measures single-attempt reliability, while larger k measures whether the model can solve the problem at all, which is relevant when best-of-n sampling or verification is available.

## Broader terms

- [Evals](https://protologue.com/t/evals/)

## Related terms

- [Best-of-N Sampling](https://protologue.com/t/best-of-n-sampling/)
- [Coding Agent](https://protologue.com/t/coding-agent/)
- [Reinforcement Learning with Verifiable Rewards](https://protologue.com/t/rlvr/)

## Sources

- Chen et al. (2021). Evaluating Large Language Models Trained on Code. https://arxiv.org/abs/2107.03374

## Cite this entry

Protologue. (2026). pass@k. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0180). https://protologue.com/t/pass-at-k/

License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)