Reassessing Muon for Matrix Factorization

Muon is a recently proposed optimizer that uses approximate orthogonalization to reshape gradient updates, showing significant performance gains in large-scale deep learning. However, it remains unclear whether these benefits are intrinsic to the optimizer or artifacts of specific deep learning settings. This paper investigates Muon's performance on simpler, well-understood problems—specifically low-rank matrix factorization—to isolate the optimizer's behavior from the complexities of modern neural network training.

The authors conduct a controlled, systematic evaluation of Muon against adaptive baselines (AdamW) and standard methods (Gradient Descent, SignGD). They utilize a variety of matrix recovery tasks, including symmetric factorization, matrix completion, and non-negative matrix factorization (NMF). A key aspect of their protocol is an exhaustive hyperparameter sweep, specifically focusing on the learning rate, to ensure that comparisons are not biased by suboptimal default configurations. They also vary problem conditioning and spectral structure to test the robustness of each optimizer.

The study finds that Muon's performance is heavily dependent on the specific problem structure. While Muon shows a consistent advantage in non-negative matrix factorization (NMF), it does not outperform AdamW on standard low-rank factorization or matrix completion tasks when both are properly tuned. The authors demonstrate that performance rankings are highly sensitive to the learning rate and that the "best" optimizer often changes depending on the conditioning and spectral distribution of the target matrix. These results suggest that Muon's success in large-scale models may be tied to specific properties of deep learning objectives rather than a universal superiority over adaptive first-order methods.