protologue

Confidence Calibration

Also called calibration, verbalized confidence, uncertainty estimation.

Confidence calibration is the degree to which a model's expressed or implied confidence matches how often it is actually right; a well-calibrated model that says it is 80% sure is correct about 80% of the time.

Description

Confidence can be read from token probabilities, by asking the model to state a confidence (verbalized confidence), or by sampling several answers and measuring agreement. Studies have found pretrained models fairly well calibrated on suitably formatted multiple-choice questions, and that for instruction-tuned models verbalized confidence can be better calibrated than token probabilities. Calibration lets an application abstain or escalate when the model is unsure. It is different from few-shot calibration, which corrects label bias in prompts.

Sources

  1. Kadavath et al. (2022). Language Models (Mostly) Know What They Know.
  2. Tian et al. (2023). Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback.

Cite this entry

Protologue. (2026). Confidence Calibration. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0121). https://protologue.com/t/confidence-calibration/

BibTeX
@misc{protologue_confidence_calibration,
  title = {Confidence Calibration},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0121},
  url = {https://protologue.com/t/confidence-calibration/}
}

Markdown JSON