protologue

Quantization

Also called model quantization, post-training quantization, PTQ, weight quantization, GPTQ, LLM.int8().

Quantization reduces the numeric precision of a model's weights, and sometimes its activations, from 16- or 32-bit floating point to lower-precision formats such as 8-bit or 4-bit integers, cutting memory use and inference cost at some cost in accuracy.

Description

Post-training quantization is applied to a model that has already been trained, without retraining it. LLM.int8() made 8-bit inference practical for large models by keeping a small number of outlier feature dimensions in higher precision. GPTQ quantizes weights to 3 or 4 bits in a single pass, using approximate second-order information to keep accuracy loss small. Quantization is widely used to fit large open-weight models onto fewer GPUs or consumer hardware. Quantization-aware training, which simulates low precision during training, is an alternative to post-training methods. Quantization is often combined with other efficiency methods such as low-rank adaptation and knowledge distillation.

Sources

  1. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.
  2. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.

Cite this entry

Protologue. (2026). Quantization. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0230). https://protologue.com/t/quantization/

BibTeX
@misc{protologue_quantization,
  title = {Quantization},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0230},
  url = {https://protologue.com/t/quantization/}
}

Markdown JSON