ContextRotBench: Measuring Long-Horizon Degradation in Coding Agents

Large language models (LLMs) are increasingly deployed as agents operating in complex, real-world software environments. However, over long-horizon trajectories, agents can exhibit context rot: the phenomenon where performance degrades as their context accumulates. Existing benchmarks frame context rot in LLMs as a needle-in-a-haystack (NIAH) problem, yet static retrieval does not translate to agentic performance, which depends on tool use, environments, and harness design. To this end, we introduce ContextRotBench, a benchmark for evaluating coding agents in realistic conditions where they are susceptible to context rot. ContextRotBench comprises 213 base tasks spanning 17 open source repositories and provides a simple framework to adapt and reuse existing SWE benchmarks; using this framework, we instantiate more than 1,000 controlled context-rot scenarios. We find that context length alone is not a primary cause of context rot; instead, degradation is driven by the content and timing of accumulated task history. Attention analysis further shows that persistent instructions receive decreasing attention despite remaining in context, suggesting a mechanism for constraint drift.