Fine-Tuning Without Melting Your GPU: A LoRA Deep Dive

The cost of full fine-tuning

Updating every weight of a multi-billion-parameter model means storing optimizer states for all of them — often three to four times the model size in memory. For most teams that is a non-starter. LoRA sidesteps it by freezing the base model and learning a small additive update.

The low-rank trick

A weight update matrix is huge but usually low-rank in practice. LoRA factors the update into two skinny matrices, A and B, so a `d x d` update becomes `d x r` times `r x d` with r as small as 8.

import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__()
        self.base = base  # frozen
        self.a = nn.Linear(base.in_features, r, bias=False)
        self.b = nn.Linear(r, base.out_features, bias=False)
        self.scale = alpha / r
        nn.init.zeros_(self.b.weight)

    def forward(self, x):
        return self.base(x) + self.scale * self.b(self.a(x))

Because B starts at zero, training begins exactly at the base model's behavior and adapts from there.

Where to place the adapters

Attention projection matrices (query and value in particular) give the most adaptation per parameter. Start there, measure, and only expand to the MLP blocks if the task needs it.

Merging for inference

At serving time you can fold `scale * B @ A` back into the base weights, so LoRA adds **zero** inference latency. Keep a separate adapter per task and hot-swap them without reloading the base — that is the real operational win.