Low-Rank Adaptation vs Fine-Tuning
Full fine-tuning updates all of a model's weights on new training data. LoRA is a parameter-efficient form of fine-tuning that freezes the original weights and trains small low-rank matrices added to them, which needs far less memory and lets several task adapters share one base model.
Definitions
- Low-Rank Adaptation PTL-0078
- Low-rank adaptation (LoRA) fine-tunes a language model by training small low-rank matrices added to its weight layers while freezing the original weights, drastically reducing the number of trainable parameters.
- Fine-Tuning PTL-0105
- Fine-tuning is the further training of a pretrained model on a smaller, task- or domain-specific dataset so that its weights adapt to that task or domain.
Differences
| Low-Rank Adaptation | Fine-Tuning | |
|---|---|---|
| Weights trained | All | A small number of added low-rank parameters |
| Memory and compute | High | Much lower |
| What you store per task | A full copy of the model | A small adapter |
Which to use
- Use LoRA for most task and style adaptation, especially with limited hardware.
- Consider full fine-tuning when adapting a model substantially and resources allow.
Sources
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models.
- Howard and Ruder (2018). Universal Language Model Fine-tuning for Text Classification.
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models.
Cite this page
Protologue. (2026). Low-Rank Adaptation vs Fine-Tuning. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0). https://protologue.com/compare/lora-vs-fine-tuning/