mvs

Everyone is fine-tuning LLMs. Almost nobody understands what is actually being updated inside the model. That distinction matters because LoRA, QLoRA, LoRA-FA, VeRA, Delta-LoRA and LoRA+ are usually discussed as if they were small variations of the same method. They are not. Some reduce trainable parameters, some reduce activation memory, some reduce the memory occupied by the frozen model, and some change the way the adapter itself is optimized. I'm actually sharing 6 techniques. 1/ LoRA Suppose a layer contains a pretrained weight matrix W. Full fine-tuning would update W directly. LoRA leaves W frozen and represents the update using two much smaller matrices, A and B, so that ΔW = BA. For a square d × d weight matrix, full fine-tuning has d² parameters available to update. A rank-r LoRA adapter has roughly 2dr trainable parameters instead, where r is normally much smaller than d. This is the basic reason LoRA can adapt very large models without training every parameter in them. 2/ LoRA-FA Standard LoRA trains both A and B. LoRA-FA freezes A and trains only B. There is a useful reason for doing this beyond simply reducing the number of trainable parameters. Computing the gradient for A requires retaining the layer input activation. If A is fixed, that gradient is no longer required, which allows LoRA-FA to reduce activation memory as the LoRA rank grows. 3/ QLoRA QLoRA attacks a different part of the memory problem. LoRA makes the adapter small, but the frozen base model can still occupy tens of gigabytes. QLoRA keeps the base model frozen in 4-bit form and trains LoRA adapters through it. The original work used NF4, double quantization and paged optimizers, and demonstrated fine-tuning a 65B model on a single 48GB GPU. This is an important distinction: QLoRA is not simply "LoRA with smaller adapters." The large memory saving comes from quantizing the frozen base model. 4/ VeRA VeRA reduces the adapter itself further. Instead of learning a separate A and B for every adapted layer, it uses frozen random low-rank matrices that can be shared across layers, while learning much smaller scaling vectors. The low-rank basis is therefore fixed. Training mainly determines how strongly different parts of that basis should contribute. This is why VeRA can use considerably fewer trainable parameters than ordinary LoRA. 5/ Delta-LoRA Ordinary LoRA treats W as fixed throughout training. Delta-LoRA relaxes that constraint. A and B are still trained, but the change in their product from one training step to the next is also used to update W. The base weights can therefore move without maintaining the ordinary gradients and optimizer states that full fine-tuning would require for W. 6/ LoRA+ LoRA+ does not introduce another adapter structure. It keeps W frozen and still trains A and B. Its change is in the optimizer: A and B use different learning rates, with B receiving a larger rate. The motivation is that the two LoRA matrices do not behave identically during optimization, so forcing them to use the same learning rate is not necessarily the best choice. Once these are separated by what they actually change, the family becomes much easier to understand. LoRA reduces the number of weights being trained. LoRA-FA also targets activation memory. QLoRA compresses the frozen base model. VeRA reduces the learned adapter parameters further. Delta-LoRA allows the pretrained weights themselves to evolve through low-rank changes, while LoRA+ keeps the LoRA structure and changes its optimization. That is really what PEFT is about => deciding which parts of a very large model actually need to move during adaptation, and which parts can remain fixed.