Test-Time Training Normalization Done Right
Zefan Cai ⋅ Hao Tan ⋅ Junjie Hu
Abstract
Chunked test-time training carries a fast weight across chunk boundaries, and \lact ends each inner-loop update by rescaling that weight row-wise to a target norm re-measured from the state itself. We observe that this measurement measures nothing: rescaling a row to the norm of the row it came from is norm-preserving, so the target recovered at every chunk is the value fixed at initialization. In exact arithmetic, re-measuring per chunk and freezing the anchor are the same map --- which rules out any modeling explanation for a difference between them in advance. In bf16 the identity fails specifically: the re-measurement reads a rounded state and feeds its error back into the state it will next measure, composing into a multiplicative random walk along the recursion. Testing the two schedules at 124M parameters in two independent batches spanning different code paths, clusters, and schedulers, we found the frozen and recompute groups fully separated on long-context loss in both batches, by $3.2\times$ and $33.8\times$ a replicate spread calibrated from a pair of mathematically equivalent arms (stratified exact permutation, one-sided $p = 1/36$). On RULER the same intervention produced no measurable improvement, which we report rather than scope around. Every cell is a single run under a hard-coded seed, so the effect --- $0.0015$ to $0.0017$ nats --- is a mechanism identified and reproduced across substrates, not a large gain. What this exposes is the \emph{time-variation} of a normalization statistic, a dimension the existing binary of ``with or without normalization'' does not separate.
Chat is not available.
Successful Page Load