Probe-and-Refine Tuning of Repository Guidance for Coding Agents

LLM-based coding agents need higher-level operational knowledge about a repository that does not exist in the code itself, which engineers supply through AGENTS.md files. Whether such guidance helps is contested: recent studies disagree on whether LLM-generated guidance improves or harms agent performance. We show that how the guidance is produced is the decisive variable, and introduce probe-and-refine tuning: synthetic bug-fix probes iteratively tune a repository's guidance file through single-shot LLM calls, with no agent loop or tool use during tuning. On SWE-bench Verified with Qwen3.5-35B-A3B (four independent trials, 200 steps), it reaches a 33.0% mean resolve rate vs. 28.3% for the static knowledge base used to initialize it and 25.5% unguided ($p < 0.001$ for probe-and-refine against both). The gain comes from coverage, not precision: refined guidance produces evaluable patches for 14.5 pp more instances while per-patch precision shows no statistically detectable difference across conditions ($\sim$56–61%, $p = 0.119$), so the intervention reshapes the agent's behavior, changing which instances it can patch, with no detectable change in how well it patches them. Guidance is also what lets the agent use a larger step budget productively, and on NVIDIA-Nemotron-3-Nano-30B-A3B the tuning loop degrades when the model cannot generate sufficiently diagnostic output, so effective guidance appears model-specific.