Reinforcement Learning from Human Feedback vs Direct Preference Optimization
Both align a model with human preferences learned from comparisons of responses. RLHF first trains a separate reward model on those comparisons and then optimizes the language model against it with reinforcement learning, while direct preference optimization trains directly on preferred-versus-rejected pairs with a simple loss, with no reward model or reinforcement learning step.
Definitions
- Reinforcement Learning from Human Feedback PTL-0011
- Reinforcement learning from human feedback (RLHF) trains a language model to produce outputs people prefer, by learning a reward model from human comparisons and optimizing the model against it.
- Direct Preference Optimization PTL-0012
- Direct preference optimization (DPO) aligns a language model to human preferences by training directly on preferred-versus-rejected response pairs, without fitting a separate reward model or running reinforcement learning.
Differences
| Reinforcement Learning from Human Feedback | Direct Preference Optimization | |
|---|---|---|
| Reward model | Required | Not needed |
| Training procedure | Reinforcement learning against the reward model | A classification-style loss on preference pairs |
| Practical trade-off | More complex; the reward model can be reused | Simpler, more stable and cheaper |
Which to use
- RLHF suits pipelines built around a reward model, which can score new samples as training proceeds.
- DPO suits teams that have preference pairs and want a simpler, cheaper training run.
Sources
- Ouyang et al. (2022). Training language models to follow instructions with human feedback.
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model.
Cite this page
Protologue. (2026). Reinforcement Learning from Human Feedback vs Direct Preference Optimization. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0). https://protologue.com/compare/rlhf-vs-dpo/