Why can an optimizer that works well on a small model become unstable as the model grows wider? We revisit common optimizers, including AdamW and Muon, through the unified perspective of steepest descent under matrix operator norms. Beyond comparing algorithms, we ask a more fundamental question: can these norms capture the underlying geometry of neural networks and explain why different optimizers behave differently as model width changes?

 A central challenge is that conventional matrix norms do not compose naturally across layers in deep networks, making it difficult to establish stability guarantees that remain independent of width. To address this, we introduce a family of mean normalized operator norms and use them to develop optimization methods better suited to wide models, including rescaled AdamW and methods based on row and column normalization. Intuitively, our goal is to identify appropriate update scales that allow each layer to remain stable as the network becomes wider. We also derive systematic learning rate scaling rules that describe how hyperparameters should change with model width, providing a principled basis for transferring hyperparameters from small models to larger ones. 

Our theory shows that the smoothness constant associated with Muon can grow as \(O(\sqrt{w})\) in the worst case, where \(w\) denotes model width, whereas our row normalization method admits a bound independent of width. Building on these insights, we introduce MOGA. In pretraining experiments with GPT2 and LLaMA at scale, MOGA, particularly its row normalization variant, achieves performance comparable to Muon while showing greater training efficiency at large token budgets and in the low loss regime.