Rethinking Architecture Search: Evaluating Neural Networks Through the Lens of Regression

The computational cost of Neural Architecture Search (NAS) has long been a bottleneck in automated machine learning. Traditional approaches require training thousands of candidate architectures on specific downstream tasks, such as CIFAR-10 or ImageNet classification, to identify high-performing designs. This task-specific paradigm assumes that architectural quality is inseparable from the target objective, yet empirical evidence suggests otherwise. Prominent architectures like ResNet and LSTM demonstrate remarkable transferability across diverse tasks and datasets, indicating that certain structural properties confer general representational power. In "Generic Neural Architecture Search via Regression," Li et al. challenge the fundamental premise of task-bound NAS, proposing GenNAS, a framework that evaluates architectures through self-supervised regression on synthetic signals rather than classification performance.

The Sample Efficiency of Regression vs. Classification

The central insight of GenNAS emerges from a critical examination of how neural networks consume training data. In classification tasks, the learning dynamics are dominated by samples near the decision boundary, while correctly classified samples contribute minimally to gradient updates. This phenomenon, rooted in the theory of support vector machines and extending to deep networks through the neural tangent kernel framework, implies that classification-based NAS wastes significant computational resources on redundant data points. The authors illustrate this geometrically; classification accuracy depends disproportionately on boundary cases, whereas regression treats all samples as equally informative.

By shifting the evaluation paradigm to regression, GenNAS exploits the full information content of the training distribution. The authors construct synthetic signal bases, combinations of sine waves and random signals, as regression targets. These synthetic tasks require the network to capture fundamental patterns such as locality, periodicity, and compositionality without collapsing representations into categorical labels. This approach preserves intermediate feature representations that classification heads typically discard, offering a more granular assessment of an architecture's capacity to transform input signals. The technical implication is profound: architecture evaluation becomes a self-supervised learning problem where the quality of learned representations, rather than task-specific accuracy, determines architectural fitness.

Synthetic Signals and the Architecture Ranking Problem

The GenNAS methodology operates on the hypothesis that an architecture's ability to regress synthetic signals correlates strongly with its performance on downstream tasks. To validate this, the authors employ Spearman's rank correlation coefficient (ρ) as the primary metric, measuring how well the synthetic regression loss predicts the true ranking of architectures on target tasks. On NAS-Bench-101, GenNAS achieves a Spearman's ρ of 0.85, compared to 0.38 for existing efficient NAS methods. This represents not merely an incremental improvement but a qualitative shift in our ability to forecast architectural performance without task-specific training.

The framework introduces an additional refinement: automatic task search. While synthetic signals provide a generic evaluation metric, the optimal combination of signal bases varies across search spaces and downstream objectives. Using only 20 architectures with known ground-truth performance on the target task, GenNAS optimizes the mixture of synthetic signals to maximize ranking correlation. This weakly-supervised approach bridges the gap between fully task-agnostic search and traditional supervised NAS, allowing the framework to adapt its evaluation criteria without requiring the thousands of fully trained models typically necessary for surrogate model construction.

Empirically, the efficiency gains are substantial. On NAS-Bench-201, GenNAS identifies near-optimal architectures within 0.3 GPU hours, compared to the tens or hundreds of GPU hours required by conventional approaches. The authors validate this efficiency across 13 CNN search spaces and one NLP space, demonstrating that the regression proxy generalizes across modalities, though with varying degrees of fidelity.

Broader Implications and Critical Analysis

The GenNAS framework suggests a decoupling of architecture search from end-to-end task optimization, raising important questions about the nature of architectural priors. If synthetic signal regression reliably identifies transferable architectures, this implies that certain structural motifs, such as residual connections or specific convolutional patterns, possess universal approximation properties that transcend task boundaries. This aligns with the observation that manually designed architectures like ResNet succeed across computer vision benchmarks, but it challenges the field to reconsider why we have defaulted to task-specific search for so long.

However, the assumption of task-agnostic architectural quality merits scrutiny. While GenNAS demonstrates strong performance on standard benchmarks, these benchmarks may not capture the full spectrum of task-specific inductive biases required for specialized applications. For instance, architectures optimized for synthetic regression may favor local connectivity patterns suitable for image data, potentially overlooking the long-range dependencies crucial for certain NLP or time-series tasks. The paper's evaluation on a single NLP space provides preliminary evidence of cross-domain transfer, but the extent to which synthetic signals capture linguistic hierarchies or sequential dependencies remains an open empirical question.

Furthermore, the reliance on synthetic signals introduces a form of inductive bias that may not align with natural data distributions. The choice of sine waves and random signals implicitly encodes assumptions about smoothness and frequency composition that characterize visual data but may not generalize to discrete or highly structured domains. There is a risk that GenNAS might prefer architectures adept at signal processing while underweighting architectures specialized for symbolic reasoning or sparse feature extraction.

Conclusion and Future Directions

The transition from classification-based to regression-based architecture evaluation represents a methodological pivot with significant practical consequences. By achieving high ranking correlation with minimal computational overhead, GenNAS addresses the scalability crisis that has limited the democratization of automated architecture design. The framework's success suggests that we may be approaching a future where "foundation architectures," analogous to foundation models, are discovered through task-agnostic criteria and subsequently adapted to specific applications.

Nevertheless, several questions remain unresolved. How does the GenNAS approach scale to the massive search spaces characteristic of Vision Transformers or modern large language models? Can the synthetic signal paradigm be extended to include architectural constraints such as hardware latency or memory footprint without sacrificing ranking accuracy? Most importantly, do the architectures discovered through synthetic regression maintain their superiority when subjected to extensive hyperparameter tuning on target tasks, or does the proxy metric break down at the extremes of performance optimization?

As the field moves toward more efficient automated machine learning systems, the work of Li et al. provides a compelling argument for rethinking our evaluation metrics. The distinction between learning to classify and learning to represent may prove to be the key distinction in the next generation of neural architecture search algorithms.