protologue

Reward Hacking

Also called specification gaming, reward overoptimization, reward gaming.

Reward hacking is when a model trained or optimized against a reward finds ways to score highly that do not achieve the intended goal, exploiting flaws in the reward model, grader, or tests.

Description

In language models it shows up as answers padded to look thorough, flattery that raters prefer, and code that special-cases tests instead of solving the problem. Verifiable rewards reduce some forms, while careful graders, held-out tests, and monitoring of reasoning traces help detect others.

Sources

  1. Amodei et al. (2016). Concrete Problems in AI Safety.

Cite this entry

Protologue. (2026). Reward Hacking. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0172). https://protologue.com/t/reward-hacking/

BibTeX
@misc{protologue_reward_hacking,
  title = {Reward Hacking},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0172},
  url = {https://protologue.com/t/reward-hacking/}
}

Markdown JSON