RankFusion: Fusing Attention and Structure Signals for Robust KV Cache Compression

KV cache eviction reduces the memory cost of long-context LLM inference by discarding unimportant key-value pairs. Existing methods rank tokens by a single scoring function. Two scoring mechanisms dominate: attention-based methods (e.g., SnapKV) measure how strongly the query attends to each token in a query-aware setting, while structure-based methods (e.g., Compactor) measure the statistical influence of each key vector on the attention distribution, independent of the query. We show that these mechanisms select largely disjoint token sets: across 1,100 samples from 6 diverse datasets, the Jaccard overlap between their top-ranked tokens is only 0.14 on average. Each mechanism leaves a blind spot the other covers—at extreme compression (95%, retaining 5% of KV pairs) on Llama-3.1-8B, SnapKV collapses on RULER (41.50) while Compactor collapses on LongReason (26.89). Building on this observation, we propose RankFusion, a family of training-free methods that combine both scoring functions at the rank level. We show that any rank-level fusion—whether equal-weight averaging (RankFusion-Avg), union selection (RankFusion-Max), or entropy-adaptive weighting (RankFusion-EA, our primary variant)—yields consistent improvements over both single-signal baselines at moderate compression, confirming that signal complementarity is the primary driver of gain. At extreme compression, where other fusion variants lose robustness, entropy-adaptive weighting provides a further advantage: RankFusion-EA is the sole variant that adapts per-head which signal to trust—achieving 51.70 on LongReason and 58.63 on RULER, consistently leading all methods across both benchmarks and model families. These results suggest that the attention-based $\times$ structure-based distinction captures a fundamental property of the token importance landscape.