protologue

Reward Model

Also called preference model, outcome reward model, ORM.

A reward model is a model trained to score a language model's outputs according to human or AI preferences, usually from pairwise comparisons, so that the score can serve as a training signal or a ranking criterion.

Description

In RLHF, people compare pairs of responses, a reward model learns to predict which one they prefer, and the language model is then optimized against it. Reward models are also used at inference time to pick the best of several sampled responses. A model that scores only the final answer is an outcome reward model, in contrast to a process reward model, which scores each reasoning step. Optimizing too hard against an imperfect reward model leads to reward hacking.

Sources

  1. Ouyang et al. (2022). Training language models to follow instructions with human feedback.
  2. Lightman et al. (2023). Let's Verify Step by Step.

Cite this entry

Protologue. (2026). Reward Model. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0106). https://protologue.com/t/reward-model/

BibTeX
@misc{protologue_reward_model,
  title = {Reward Model},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0106},
  url = {https://protologue.com/t/reward-model/}
}

Markdown JSON