Why the Bases Matter More Than the Rank in Vector-Based Adaptation

Parameter-efficient fine-tuning (PEFT) has become the default way to specialize large models without paying the full cost of updating every weight. LoRA is the method most practitioners reach for, but a quieter line of work, vector-based adaptation, pushes the parameter count even lower. The catch has always been that these methods need much larger ranks to match LoRA, which erodes the very efficiency that makes them attractive. A recent paper, GiVA: Gradient-Informed Bases for Vector-Based Adaptation by Gangwar et al. (AISTATS 2026), argues that the problem was never the rank itself. It was the choice of bases. This post unpacks that claim and considers what it tells us about the geometry of fine-tuning.

The setup: what vector-based adaptation actually does

To appreciate GiVA, it helps to recall the structure shared across LoRA and its descendants. LoRA models the weight update as a product of two low-rank matrices, $\Delta W = BA$, where $B \in \mathbb{R}^{m \times r}$ and $A \in \mathbb{R}^{r \times d}$. Both matrices are trained, and the trainable parameter count scales with $r(m + d)$.

Vector-based adaptation methods take a more aggressive position. They freeze the two matrices, calling them bases, and insert small trainable scaling vectors between them. The adapted update takes the form $\hat{B},\Gamma,\hat{A},\Lambda$, where only the diagonal scaling vectors $\Lambda$ and $\Gamma$ are learned. Because you train only vectors rather than full matrices, the trainable parameter count drops to something on the order of $r + m$, far below LoRA. This extreme economy is appealing in federated learning, multi-device fine-tuning where updates travel over a network, and mixture-of-experts systems where you want many lightweight specializations.

The two prior methods the paper compares against differ in how they pick the frozen bases:

• VeRA uses a pair of random matrices, shared across all layers, drawn from a Kaiming distribution.
• OSoRA derives its bases from the singular value decomposition of the pre-trained weights $W_{pt}$.

Both share a critical weakness. The bases are fixed and contain no information about the downstream task. VeRA's bases are random; OSoRA's encode only what the pre-trained model already knows. With the bases frozen and task-agnostic, the only way to recover enough expressive capacity is to crank up the rank. The paper reports a concrete consequence: fine-tuning Qwen 2 (0.5B) on 15K commonsense reasoning examples with VeRA takes roughly 2.5 times LoRA's runtime, driven by a rank of 1024 against LoRA's 16. The parameter savings are real, but the wall-clock cost is not.

The GiVA idea: align the first-step gradient

GiVA's contribution is a different basis initialization. Rather than using random matrices or the SVD of the pre-trained weights, it uses the SVD of the first-step full fine-tuning gradient $\nabla_W \mathcal{L}(W_{pt})$. Concretely, it sets $A \leftarrow V_r^T$ from that decomposition, constrains $B$ to be orthonormal ($B^T B = I_{r \times r}$), initializes $\Lambda \leftarrow 1$ and $\Gamma \leftarrow 0$, and keeps the residual at the pre-trained weights.

The principle is that the bases should be chosen so that the first-step gradient of GiVA closely approximates the first-step gradient of full fine-tuning. In other words, instead of giving the scaling vectors an arbitrary subspace to operate in, you hand them the subspace that the optimization signal itself points toward at step one. The scaling vectors then have to do far less work to reach a useful update, because the directions they modulate already align with where the loss wants to move.

This is conceptually related to recent improvements to LoRA itself, such as gradient-aware or SVD-based initializations (the paper cites Wang et al., 2024, 2025). The distinguishing feature of GiVA is that it imports this gradient-informed initialization into the vector-based setting, where only the scaling vectors are trained. You get the directional advantage of a gradient-aligned subspace while preserving the extreme parameter economy that defines vector-based adaptation.

The headline result and what it costs

The empirical claim is the part worth scrutinizing. Across natural language understanding, natural language generation, and image classification, GiVA reportedly matches or beats existing vector-based methods and LoRA while reducing rank requirements by a factor of eight. Because the rank drops by 8x, the training time falls back into line with LoRA, eliminating the runtime penalty that made VeRA and OSoRA less practical despite their parameter savings.

It is worth being precise about what is and is not free here. The gradient-informed initialization requires computing the first-step full fine-tuning gradient before training begins. That is a one-time forward and backward pass over a batch with the full model in memory, which is more expensive than VeRA's trivial random initialization. For very large models, the memory footprint of that single full gradient computation is non-trivial, even if amortized over the entire training run it is small. The paper also explores three alternative basis initialization strategies and finds they perform similarly, which suggests the method is not delicately dependent on one specific construction. That robustness is reassuring, but it also raises a question I return to below.

My take: this reframes the efficiency trade-off

The result that I find most interesting is not the 8x rank reduction itself but what it implies about where capacity lives in these methods. The conventional reading of vector-based adaptation is that you trade rank for parameter count: pay more rank to recover the expressiveness you lose by freezing the matrices. GiVA shows that this trade is partly an artifact of poor initialization. The frozen bases were squandering their dimensions on directions irrelevant to the task. Once those dimensions point in informative directions, you need far fewer of them.

There is a clean intuition here that connects to the broader low-rank fine-tuning literature. Aghajanyan et al. (2021) showed that task adaptation lives in a low-dimensional subspace. The trouble with random or pre-trained-weight bases is that there is no reason for that subspace to coincide with the span of the random or pre-trained directions, so you need a large rank to ensure the relevant subspace is contained inside what you froze. A gradient-informed basis essentially rotates the frozen subspace to overlap with the task subspace, which is why a small rank now suffices.

That framing also exposes a limitation. The bases are chosen from the gradient at the pre-trained point only. The relevant subspace can drift as training proceeds, and a first-step gradient is a local, possibly noisy estimate of where the loss surface curves. The fact that three different basis strategies give similar results hints that the method may be operating in a regime where the precise subspace matters less than simply not wasting it on irrelevant directions. If so, GiVA's advantage over VeRA may come as much from coarse alignment as from fine-grained optimality, which would suggest there is still room to improve.

Open questions and forward look

Several threads seem worth pursuing. First, does a gradient-informed basis computed at a later checkpoint, or refreshed periodically, buy additional rank reduction? Second, how does GiVA behave under the very settings that motivate vector-based adaptation, namely federated and mixture-of-experts deployments, where the communication and storage savings matter most and where the one-time gradient cost may be harder to amortize? Third, the relationship between basis informativeness and the noise of the first-step gradient deserves a careful study; a single batch may not be a stable estimate of the task subspace for small datasets.

The broader lesson, as the accompanying summary put it, is that where you start matters more than how wide you go. For a field that has spent considerable effort tuning ranks and adapter widths, GiVA is a useful reminder that initialization geometry can be the dominant variable. If the result holds across the larger models and noisier tasks that practitioners actually face, vector-based adaptation may finally deliver its parameter savings without the runtime tax that has held it back.