Leyline: KV-Cache Edit Directives for Efficient Agentic Reasoning
Bole Ma ⋅ Jan Eitzinger ⋅ Harald Koestler ⋅ Gerhard Wellein
Abstract
Reasoning agents rewrite their own context as they work: they drop failed tool calls, truncate stale output, and summarize old turns. Serving systems, however, assume context only ever grows, so every mid-context edit silently discards the cached attention state (the KV cache) for everything after the edit and recomputes it from scratch, a tax paid on every turn of a long trajectory. Serving infrastructure is already reaching for an agent-to-cache interface like NVIDIA Dynamo's agent hints, but every verb in that channel manages the reuse of immutable cache blocks; none can express an edit. We introduce Leyline, a serving-side primitive that supplies the missing verb. An agent attaches a small directive to its request: replace this span with this stub, and either keep or erase the removed content's influence on the rest of the cache. The serving stack applies the edit in place: the text before the edit keeps its cache, the replacement is computed fresh, and everything after is repaired with one closed-form rotation per cached entry instead of a recomputation. The repair is load-bearing (removing it degrades output fidelity $68\times$ on real agent contexts), holds across open weights models spanning two attention families, and beats or matches recomputation-based alternatives at equal budget. On a live SGLang server the primitive raises cache hit rate by 11 points and cuts median latency up to 13.3\% at high concurrency, and a ten-line context-truncation policy routed through the same interface raises an agent's debugging success rate from 31\% to 45\%. Because every edit is declared, the interface also gives the silent context rewriting that agent frameworks already perform an explicit, auditable form.
Chat is not available.
Successful Page Load