protologue

Evaluation Awareness

Also called eval awareness.

Evaluation awareness is a language model's ability to tell whether it is being evaluated or used in real deployment, which can make its behavior during tests unrepresentative of its behavior in use.

Description

In the introducing study, frontier models classified 1,000 transcripts drawn from 61 datasets as evaluation or deployment above chance but below a human baseline, with the best model reaching an AUC of 0.83 against 0.92 for humans. They did better on agentic transcripts than on chat, and could often tell whether an evaluation tested capability or alignment. Later work found that evaluation awareness is linearly decodable from the internal activations of every open-weight model tested, even though models rarely say they are being evaluated, and that steering along that direction increases how often they say so. The authors recommend measuring the internal representation, its verbalization and its sensitivity to steering separately. Evaluation awareness matters because safety and alignment evaluations assume a model behaves the same whether or not it is observed.

Sources

  1. Needham et al. (2025). Large Language Models Often Know When They Are Being Evaluated.
  2. Heidari et al. (2026). Evaluation Awareness in Language Models: Representation, Verbalization, and Control.

Cite this entry

Protologue. (2026). Evaluation Awareness. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0227). https://protologue.com/t/evaluation-awareness/

BibTeX
@misc{protologue_evaluation_awareness,
  title = {Evaluation Awareness},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0227},
  url = {https://protologue.com/t/evaluation-awareness/}
}

Markdown JSON