When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?

Speculative decoding accelerates large language model inference, yet recent hidden-state-based drafters such as EAGLE3 and MTP exhibit long-range decay: draft accuracy drops as the speculative horizon increases. While prior work attributes this issue to train–inference mismatch and mitigates it with autoregressive test-time training (TTT), we find that the decay persists even after TTT. We revisit this phenomenon through the lens of context preservation. We argue that reusing target hidden states provides only a biased context compression: the hidden state summarizes history according to the current attention query and is therefore optimized for immediate next-token prediction, potentially discarding information needed by later speculative steps. In contrast, the target KV cache preserves token-wise contextual representations explicitly. This motivates the KV-Reuse Hypothesis: reusing the target KV cache can provide richer conditioning signals for long-horizon drafting. To evaluate this hypothesis, we introduce KVShot, a diagnostic framework that compares three reuse paradigms: hidden-only, KV-only, and hybrid. Experiments on Qwen3-8B show that KV reuse improves long-range acceptance, although current training pipelines yield only marginal end-to-end speedups. Further analysis reveals two structural bottlenecks: shallow drafters struggle to estimate target queries, and draft-side KV projections receive sparse gradient signals. These results suggest that fully realizing KV-aware speculative decoding requires moving beyond autoregressive TTT toward block-wise training. KVShot thus provides both a diagnostic testbed and empirical guidance for designing next-generation speculative inference architectures.