Harmonic: A Hierarchical State Space Model with Multi-Timescale Recurrence

We present Harmonic, a hierarchical state space model (SSM) for language modeling. The architecture stacks three recurrent levels at progressively slower timescales; each level receives the prediction error of the level below as input, rather than its raw hidden state. On enwiki8 with equal token budgets, Harmonic outperforms a comparable Transformer (28M params) by +1.4% at 1K tokens, +6.7% at 8K tokens, and +11.4% at 32K tokens (bpt, lower is better). It also outperforms Mamba at every tested length by 0.7--1.8%. At 64K tokens, both Mamba and Transformer run out of memory on an 80GB H100; Harmonic trains successfully, reaching 6.169 bpt — a direct consequence of O(L) memory. Results replicate on WikiText-103 (H–TF gap +1.7% to +7.2% across 1K–32K). Compute is O(L) per forward pass vs. O(L²) for attention.