protologue

Speculative Decoding

Also called speculative sampling, assisted generation.

Speculative decoding is an inference technique in which a smaller, faster draft model proposes several next tokens and the large target model checks them in a single parallel pass, accepting or correcting them so that generation is faster while the output distribution stays the same as sampling from the target model alone.

Description

Because the target model verifies several drafted tokens per forward pass instead of generating one token per pass, decoding needs fewer sequential calls to the large model. A sampling rule that accepts or rejects each drafted token is what preserves the target model's distribution. The speedup depends on how often the draft model's guesses are accepted. A closely related method was published shortly afterwards under the name speculative sampling.

Sources

  1. Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding.
  2. Chen et al. (2023). Accelerating Large Language Model Decoding with Speculative Sampling.

Cite this entry

Protologue. (2026). Speculative Decoding. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0217). https://protologue.com/t/speculative-decoding/

BibTeX
@misc{protologue_speculative_decoding,
  title = {Speculative Decoding},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0217},
  url = {https://protologue.com/t/speculative-decoding/}
}

Markdown JSON