FasterWeight-MLP Attention: Rethinking Sequence Modeling from a Neural-Network-Native Perspective

Linear-time attention variants avoid the quadratic score matrix by adding gates, decays, or prescribed update rules motivated by retrieval, regression, or optimization interpretations. We instead view the object optimized by data as the learnable function class that generates causal mixing weights, and introduce FasterWeight-MLP Attention (FW-MLP), a nested fast-weight parameterization of the three causal regions induced by a query--context pair $(t,i)$: the query prefix $X_{1:t}$, the key prefix $X_{1:i}$, and the between-boundary interval $X_{i+1:t}$. Endpoint factors are asymmetric prefix-instantiated fast MLPs; the full model adds an exactly cancellative interval fast MLP whose exponential tracks admit a linear-time recurrence. Our theory characterizes the bilinear limits of raw linear attention, the approximation power of endpoint fast MLPs, exact interval cancellation, and tri-region separability/density. The interval recurrence is linear-time in sequence length, and this is not only asymptotic: measured forward-pass throughput is on par with recent linear-time attention variants ($1.03\times$ their median on an A100). Empirically, FW-MLP obtains the best aggregate rank on VariantBench, a clear gain on an interval-dependent retrieval diagnostic, the best matched-budget nanoGPT validation loss, and the best average downstream rank in unified 340M/1.3B language-model evaluations.