Gating Mechanisms for Continually Adapting Compressed KV Memory

Serving large language models over long contexts is expensive because cost scales with the key–value (KV) cache, and accuracy degrades when relevant evidence is buried among distracting tokens. Compressed memories address this problem by distilling a long context into a small, trainable KV cache. Cartridges learn such a cache through a synthetic question-answering objective called self-study. Real deployments, however, are not static: new corpora arrive continually, and a memory with a fixed budget must absorb each one while retaining what it already holds. We study this problem through gating mechanisms that govern what enters a compressed memory and where it is stored. We introduce two mechanisms for the dense Cartridge: a statistical TF–IDF gate that updates only the slots attended to by a new corpus but not by an earlier one, and a learned mixture-of-experts gate that routes each query to a small subset of expert sub-caches. Across two instruction-tuned models (Llama-3.2-3B and Qwen3-4B) in a continual setting, the best routed configurations improve both forgetting and acquisition relative to a dense baseline of equal capacity. However, router analysis shows little corpus-specific specialization, suggesting that routing helps mainly by limiting which parts of the cache can change during an update.