protologue

pass@k

Also called pass@1, pass at k.

pass@k is an evaluation metric for code generation, and other tasks with automatic checks, defined as the probability that at least one of k sampled solutions to a problem passes its tests.

Description

It was popularized with an unbiased estimator computed from n ≥ k samples per problem. pass@1 measures single-attempt reliability, while larger k measures whether the model can solve the problem at all, which is relevant when best-of-n sampling or verification is available.

Sources

  1. Chen et al. (2021). Evaluating Large Language Models Trained on Code.

Cite this entry

Protologue. (2026). pass@k. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0180). https://protologue.com/t/pass-at-k/

BibTeX
@misc{protologue_pass_at_k,
  title = {pass@k},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0180},
  url = {https://protologue.com/t/pass-at-k/}
}

Markdown JSON