Rethinking the Default: Why Muon Deserves Attention in Tabular Deep Learning

Tabular deep learning occupies a peculiar position in the machine learning landscape. While computer vision and natural language processing have witnessed dramatic architectural innovations, supervised learning on tabular data remains stubbornly reliant on multi layer perceptrons (MLPs). These architectures, despite their simplicity, form the backbone of the strongest contemporary methods, from TabR to TabPFN variants. Yet, as the recent comprehensive study by Gorishniy, Rubachev, Feoktistov, and Babenko titled "Benchmarking Optimizers for MLPs in Tabular Deep Learning" demonstrates, the community may have overlooked a critical dimension of model performance: the optimization algorithm itself.

For years, AdamW has served as the unquestioned default for training tabular neural networks. This preference, while pragmatic, has persisted without systematic validation against newer alternatives. The researchers challenge this complacency by evaluating 15 distinct optimizers across 17 datasets, revealing that algorithmic choices in optimization can substantially alter generalization performance.

The Muon Advantage

The central finding of the benchmark is the consistent superiority of the Muon optimizer over AdamW. Muon, originally introduced by Jordan and colleagues, diverges fundamentally from adaptive gradient methods. Rather than maintaining per parameter second moment estimates like AdamW, Muon applies an orthogonalization procedure to the momentum buffer using Newton Schulz iterations. This enforces a form of spectral normalization on the updates, effectively decoupling the step size from the directional components of the gradient.

In the context of tabular data, where feature spaces are often heterogeneous and high dimensional relative to sample sizes, this geometric approach to optimization appears particularly effective. The authors report that Muon achieves better test set performance across both vanilla MLPs and modern architectural variants. This consistency is noteworthy because it suggests the optimizer is robust to architectural changes, from simple feed forward networks to more complex residual structures with normalization layers.

However, this performance comes with a computational price. The study documents an average slowdown of 3× during hyperparameter tuning. This overhead stems primarily from the Newton Schulz iterations required to orthogonalize the momentum matrices. For practitioners operating under tight computational budgets, this presents a genuine tradeoff. Yet for applications where model performance is paramount, the cost may be justified, particularly since the overhead affects tuning rather than inference.

The EMA Complement

Beyond the headline comparison between Muon and AdamW, the paper identifies a subtler but practical technique: exponential moving average (EMA) of model weights. When applied to vanilla MLPs trained with AdamW, EMA consistently improves generalization. The mechanism here is well understood in other domains; EMA stabilizes the optimization trajectory by maintaining a shadow copy of parameters that averages out the noise in stochastic gradients.

Interestingly, this benefit attenuates when moving to more sophisticated architectures. For modern MLP based methods incorporating advanced regularization techniques, EMA provides inconsistent improvements. This suggests an interaction effect between architectural inductive biases and optimization stability. As architectures become more complex, perhaps incorporating their own forms of implicit regularization, the marginal benefit of EMA diminishes. This finding serves as a caution against universally applying techniques that work well on baselines to more advanced systems.

Methodological Rigor

The experimental design merits attention because it addresses domain specific characteristics often ignored in broader optimizer benchmarks. Unlike language model pretraining or computer vision tasks where training proceeds for a fixed budget, tabular deep learning typically employs early stopping based on validation metrics. The final evaluation occurs on a held out test set, emphasizing generalization over convergence speed.

This distinction is crucial. Many optimizer benchmarks prioritize time to target loss or validation accuracy, rewarding methods that make rapid initial progress. In tabular settings, however, aggressive early optimization can lead to overfitting on small datasets. By standardizing early stopping protocols and focusing on test set performance, the authors ensure their conclusions apply to the actual use case of tabular practitioners.

Original Insights and Implications

The implications of this study extend beyond a simple recommendation to switch optimizers. They illuminate a broader pattern in deep learning research: the tendency to focus on architectural innovation while treating optimization as a solved problem. The tabular community has spent years refining MLP architectures, adding residual connections, attention mechanisms, and sophisticated embedding layers, all while using the same optimization recipe. That a change in optimizer can yield consistent improvements suggests there is untapped headroom in existing architectures.

The Muon result also highlights the geometric peculiarities of tabular data. Unlike images or text, where local structure and sequential dependencies dominate, tabular features often exhibit complex interactions and varying scales. Muons explicit handling of the update geometry through orthogonalization may better respect the natural metric structure of these parameter spaces. One might speculate that the curvature of loss landscapes in tabular learning differs systematically from other domains, making naive adaptive methods suboptimal.

The cost analysis, however, introduces a sobering note. A 3× increase in tuning time is not trivial, especially in production environments where model iteration cycles matter. This creates a bifurcation in best practices: researchers seeking state of the art results on benchmark datasets should adopt Muon, while practitioners requiring rapid prototyping might stick with AdamW paired with EMA for vanilla MLPs. Future work might investigate approximations to Muon that reduce the Newton Schulz overhead without sacrificing the geometric benefits.

Furthermore, the inconsistent EMA results for advanced architectures suggest we need a more nuanced understanding of how regularization techniques compose. Modern tabular models already incorporate dropout, stochastic depth, and various normalization schemes. Adding EMA to this mix does not yield additive benefits, implying potential redundancy or interference between stabilization mechanisms. This calls for a systematic study of regularization stacking in tabular contexts.

Conclusion

The benchmark by Gorishniy and colleagues serves as a necessary corrective to the inertia surrounding optimizer selection in tabular deep learning. By demonstrating that Muon consistently outperforms AdamW across diverse datasets and architectures, they provide empirical justification for accepting higher computational costs in pursuit of better generalization. The finding that EMA improves simpler but not more complex models adds nuance to our understanding of optimization stability.

Looking forward, several questions remain open. Can the computational overhead of Muon be amortized through better implementation or approximation? Do these findings extend to the emerging domain of tabular foundation models and in context learning? And perhaps most importantly