protologue

Evals

Also called evaluations, LLM evaluation, eval suite, test set.

Evals are systematic tests of a model or LLM application, consisting of a set of representative inputs and a grading method, used to measure performance against defined success criteria and to compare prompts, models, and changes.

Description

Grading can be exact-match or code-based, done by an LLM judge with a rubric, or done by people. Code-based grading is fastest and most reliable where it applies. A good eval set mirrors real usage, including edge cases, and turns prompt engineering from guesswork into measurement. Without one, changes that fix one case can silently break others.

Sources

  1. Anthropic (2024). Define success criteria and build evaluations.
  2. Liang et al. (2022). Holistic Evaluation of Language Models.

Cite this entry

Protologue. (2026). Evals. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0179). https://protologue.com/t/evals/

BibTeX
@misc{protologue_evals,
  title = {Evals},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0179},
  url = {https://protologue.com/t/evals/}
}

Markdown JSON