Recurrent Key-Value Residual

Long sequence modeling requires preserving information across distant tokens, views, or video chunks while still refining that information through deep Transformer computation. Standard attention provides an explicit key-value memory, but its cost grows with context length, while compressed recurrent states can discard information needed by later layers. We propose recurrent Key-Value residual, a recurrent Transformer interface that injects the top-layer key and value state from the previous recurrent step into the key and value projections of every layer in the current step. Unlike standard value residual connections, which reuse shallow first-layer value features within the same segment, our mechanism carries mature high-level representations across both depth and time. The key residual preserves an addressable retrieval interface, while the value residual propagates content produced after the full Transformer stack has processed the previous segment. We instantiate this idea in a dual-stream recurrent Transformer, where a local stream models short-range interactions under a bounded attention window and a recurrent stream propagates long-range state across segments. We evaluate the architecture on novel view synthesis, autoregressive video generation, and long-context language modeling, showing that recurrent KV residual provides a simple and general interface for extending Transformer memory without expanding the active attention window.