Decoupling Regression Capacity from Model Scale: A Technical Analysis of RELISH

The text to text paradigm has unified natural language processing under a single generative framework. Yet this unification comes with significant costs when applied to predictive tasks like regression. In their preprint "LLM REgression with a Latent Iterative State Head" (RELISH), Yiheng Su and Matthew Lease identify a fundamental architectural tension. When large language models predict continuous values, they face a representational mismatch. Language modeling objectives penalize token mismatches discretely, whereas regression requires optimizing for numerical proximity in continuous space. A prediction of 0.9 incurs the same cross entropy penalty as 0.1 when the target is 1.0, despite being numerically closer. Su and Lease argue that this semantic gap necessitates architectural innovations that bypass text generation entirely while preserving the rich representations learned by LLMs.

The Architecture of Iterative Refinement

Existing approaches to LLM based regression fall into three families, each with distinct trade offs. Autoregressive decoding prompts the model to generate numeric strings, remaining faithful to pretraining but sacrificing numerical calibration. Regression aware inference methods, such as RAFT, apply Bayes optimal decision rules to the output distribution to minimize expected error, but require multiple sampling passes or enumeration of candidates at inference time. Predictive head methods offer the efficiency of single pass inference by attaching task specific regressors to frozen hidden states, yet historically underperform because they rely on simple pooling heuristics like mean aggregation over token representations.

RELISH occupies a unique position within the predictive head family. Rather than collapsing token representations through static pooling, the architecture introduces a learned latent state that undergoes iterative refinement via cross attention over the LLM's token level outputs. This latent state functions as a dynamic query vector that progressively reweights which dimensions of the input sequence matter for the scalar prediction. After several iterations of refinement, the final state maps to a point estimate through a linear regressor. The mechanism acknowledges that regression relevant information may be distributed across noncontiguous token spans or hierarchical structures within the hidden states. Static pooling assumes uniform importance across positions, whereas RELISH allows the model to learn complex attention patterns that emphasize numerically salient features.

The parameter efficiency of this approach is striking. The architecture requires approximately 3.4 million to 3.7 million trainable parameters regardless of whether the backbone contains 8 billion or 32 billion parameters. This represents only 0.01 percent to 0.04 percent additional overhead. In contrast, LoRA based alternatives scale with model width, consuming 0.26 percent to 0.42 percent of backbone parameters. This decoupling of regression capacity from model size challenges the prevailing assumption that task specific adapters must grow proportionally with foundation model scale.

Empirical Results and Comparative Analysis

Su and Lease evaluate RELISH across five datasets spanning semantic textual similarity (STS) and machine translation quality estimation, using four LLM backbones and both frozen and LoRA fine tuned training regimes. The results demonstrate consistent superiority over all three competing families. Against RAFT, which represents the current leading regression aware approach, RELISH achieves a 6.7 percent improvement in Pearson correlation, a 7.2 percent improvement in Spearman correlation, and a 19.9 percent reduction in range normalized RMSE.

These gains prove particularly notable given the computational asymmetry between methods. RAFT and similar regression aware techniques require approximating posterior distributions through multiple forward passes or candidate enumeration. RELISH maintains single pass inference while outperforming these computationally heavier alternatives. The performance advantage holds across both frozen and fine tuned backbone regimes, suggesting that the bottleneck in regression accuracy lies not in the quality of the underlying representations but in the mechanism for extracting scalar predictions from them.

The evaluation also reveals interesting patterns regarding model scale. While autoregressive methods show erratic scaling behavior and regression aware methods become increasingly expensive as model size grows, RELISH maintains constant inference cost and consistent relative improvements. This stability indicates that the iterative latent state successfully distills regression relevant information without requiring additional capacity from the backbone as it scales.

Original Insights and Limitations

The RELISH architecture illuminates an underexplored dimension in LLM adaptation theory. Most parameter efficient fine tuning literature assumes that adaptation complexity should correlate with model size, positing that larger models require more expressive adapters to steer their behavior. RELISH demonstrates that for regression tasks, a fixed size iterative mechanism suffices regardless of backbone scale. This suggests that the information necessary for scalar prediction is already present in frozen representations; the challenge lies in extraction rather than modification.

However, the approach carries limitations that warrant honest scrutiny. The current formulation handles only single scalar outputs. Extending to multidimensional regression or structured prediction would require architectural modifications, potentially including multiple latent states or output heads. Additionally, the iterative refinement assumes that cross attention over frozen representations captures all necessary reasoning for numerical prediction. For tasks requiring arithmetic operations or complex numerical relationships not explicitly encoded in the pretraining distribution, deeper integration between the latent state and external computational tools might become necessary.

The comparison between frozen and LoRA training regimes also raises questions about where regression relevant information resides. RELISH maintains advantages in both settings, which implies that full fine tuning may be unnecessary for many regression tasks. Yet this finding also suggests that current LLMs may already encode sufficient semantic information for scalar prediction tasks, and that the field's recent emphasis on fine tuning for every downstream task may be misallocated effort for predictive applications.

Conclusion and Future Directions

RELISH points toward a future where LLM adaptation is task specialized rather than scale dependent. The iterative latent state mechanism offers a template for other structured prediction challenges beyond regression, potentially including ranking, scoring, and continuous control tasks. The demonstration that 3.4 million parameters suffice to outperform methods using orders of magnitude more trainable capacity invites the field to reconsider whether bigger adapters are always better.

Several open questions remain. Can similar decoupling strategies work for classification tasks with large label spaces, or does the discrete nature of such tasks inherently require scale dependent adaptation? Would incorporating explicit numerical computation modules into the iterative refinement process improve performance on arithmetic reasoning tasks? Most importantly, does the success of RELISH indicate that LLM representations are more task agnostic than previously assumed, or does it simply reflect the particular structure of text based regression tasks?

As foundation models continue to grow, architectural innovations that extract maximum utility from frozen parameters become increasingly valuable. RELISH represents a significant step in this direction, proving that with the right extraction mechanism, smaller can indeed be more effective.