Does Efficient Attention Change the Scaling Law, or Just Shift It?
Billy Dickson ⋅ Zoran Tiganj
Abstract
On a controlled 60M-370M ladder ($\le 8\times10^{19}$ FLOPs per run) where only the attention mechanism varies, full softmax, sliding-window, and linear (Gated DeltaNet) attention share the same loss-compute exponent to within bootstrap uncertainty: every 95\% confidence interval on the exponent difference $\Delta\alpha$ contains zero. The families separate in the offset of the loss curve. We additionally evaluate long-context retrieval and find that sliding-window attention, despite matching full softmax on loss, collapses past its window.
Chat is not available.
Successful Page Load