# Confidence Calibration

> Confidence calibration is the degree to which a model's expressed or implied confidence matches how often it is actually right; a well-calibrated model that says it is 80% sure is correct about 80% of the time.

- Identifier: PTL-0121
- Category: Self-Critique & Verification
- Canonical URL: https://protologue.com/t/confidence-calibration/
- Also known as: calibration, verbalized confidence, uncertainty estimation

## Description

Confidence can be read from token probabilities, by asking the model to state a confidence (verbalized confidence), or by sampling several answers and measuring agreement. Studies have found pretrained models fairly well calibrated on suitably formatted multiple-choice questions, and that for instruction-tuned models verbalized confidence can be better calibrated than token probabilities. Calibration lets an application abstain or escalate when the model is unsure. It is different from few-shot calibration, which corrects label bias in prompts.

## Related terms

- [Log Probabilities](https://protologue.com/t/log-probabilities/)
- [Hallucination](https://protologue.com/t/hallucination/)
- [SelfCheckGPT](https://protologue.com/t/selfcheckgpt/)
- [Few-shot Calibration](https://protologue.com/t/few-shot-calibration/)

## Sources

- Kadavath et al. (2022). Language Models (Mostly) Know What They Know. https://arxiv.org/abs/2207.05221
- Tian et al. (2023). Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. https://arxiv.org/abs/2305.14975

## Cite this entry

Protologue. (2026). Confidence Calibration. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0121). https://protologue.com/t/confidence-calibration/

License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)