# Quantization

> Quantization reduces the numeric precision of a model's weights, and sometimes its activations, from 16- or 32-bit floating point to lower-precision formats such as 8-bit or 4-bit integers, cutting memory use and inference cost at some cost in accuracy.

- Identifier: PTL-0230
- Category: Prompt Optimization
- Canonical URL: https://protologue.com/t/quantization/
- Also known as: model quantization, post-training quantization, PTQ, weight quantization, GPTQ, LLM.int8()
- Introduced: 2022

## Description

Post-training quantization is applied to a model that has already been trained, without retraining it. LLM.int8() made 8-bit inference practical for large models by keeping a small number of outlier feature dimensions in higher precision. GPTQ quantizes weights to 3 or 4 bits in a single pass, using approximate second-order information to keep accuracy loss small. Quantization is widely used to fit large open-weight models onto fewer GPUs or consumer hardware. Quantization-aware training, which simulates low precision during training, is an alternative to post-training methods. Quantization is often combined with other efficiency methods such as low-rank adaptation and knowledge distillation.

## Related terms

- [Knowledge Distillation](https://protologue.com/t/knowledge-distillation/)
- [Low-Rank Adaptation](https://protologue.com/t/low-rank-adaptation/)
- [KV Cache](https://protologue.com/t/kv-cache/)
- [Speculative Decoding](https://protologue.com/t/speculative-decoding/)

## Sources

- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. https://arxiv.org/abs/2208.07339
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. https://arxiv.org/abs/2210.17323

## Cite this entry

Protologue. (2026). Quantization. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0230). https://protologue.com/t/quantization/

License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)