Pretraining Data Scale Reverses the AdamW–Muon Fine-Tuning Gap

Muon has recently emerged as a promising optimizer for pretraining large language-models, offering a speedup over AdamW at minimizing pretraining loss. However, it remains unclear if Muon's advantage persists after fine-tuning. In this paper, we empirically study the downstream behaviors of Muon and AdamW, and uncover a surprising trend; beyond a sufficiently large pretrain token budget, AdamW outperforms Muon after fine-tuning. We complement our empirical findings with a controlled study of Muon and AdamW in a toy linear associative memory setting where we show that Muon learns geometrically smaller margins than AdamW, leaving examples more sensitive to weight updates that serve as a proxy for fine-tuning induced updates.