# Reinforcement Learning from Human Feedback vs Direct Preference Optimization

> Both align a model with human preferences learned from comparisons of responses. RLHF first trains a separate reward model on those comparisons and then optimizes the language model against it with reinforcement learning, while direct preference optimization trains directly on preferred-versus-rejected pairs with a simple loss, with no reward model or reinforcement learning step.

- Canonical URL: https://protologue.com/compare/rlhf-vs-dpo/

## Reinforcement Learning from Human Feedback

Reinforcement learning from human feedback (RLHF) trains a language model to produce outputs people prefer, by learning a reward model from human comparisons and optimizing the model against it.
Entry: https://protologue.com/t/rlhf/

## Direct Preference Optimization

Direct preference optimization (DPO) aligns a language model to human preferences by training directly on preferred-versus-rejected response pairs, without fitting a separate reward model or running reinforcement learning.
Entry: https://protologue.com/t/direct-preference-optimization/

## Differences

| | Reinforcement Learning from Human Feedback | Direct Preference Optimization |
|---|---|---|
| Reward model | Required | Not needed |
| Training procedure | Reinforcement learning against the reward model | A classification-style loss on preference pairs |
| Practical trade-off | More complex; the reward model can be reused | Simpler, more stable and cheaper |

## Which to use

- RLHF suits pipelines built around a reward model, which can score new samples as training proceeds.
- DPO suits teams that have preference pairs and want a simpler, cheaper training run.

## Sources

- Ouyang et al. (2022). Training language models to follow instructions with human feedback. https://arxiv.org/abs/2203.02155
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. https://arxiv.org/abs/2305.18290

## Cite this page

Protologue. (2026). Reinforcement Learning from Human Feedback vs Direct Preference Optimization. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0). https://protologue.com/compare/rlhf-vs-dpo/

License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)