protologue

Group Relative Policy Optimization

Also called GRPO.

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm for language models, a variant of proximal policy optimization, that samples a group of completions for each prompt and uses the group's average reward as the baseline for each completion's advantage instead of a learned value model.

Description

Dropping the separate value model that PPO trains alongside the policy reduces the memory and compute needed for training. In common implementations each completion's advantage is its reward minus the group mean, divided by the group's standard deviation, optionally with a KL penalty toward a reference model. GRPO was introduced for mathematical reasoning and is widely used for reinforcement learning with verifiable rewards when training reasoning models.

Sources

  1. Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.
  2. Hugging Face (2025). GRPO Trainer.

Cite this entry

Protologue. (2026). Group Relative Policy Optimization. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0216). https://protologue.com/t/group-relative-policy-optimization/

BibTeX
@misc{protologue_group_relative_policy_optimization,
  title = {Group Relative Policy Optimization},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0216},
  url = {https://protologue.com/t/group-relative-policy-optimization/}
}

Markdown JSON