Fixing the Step, Missing the Answer: What Accuracy Cannot See in Reasoning Repair
Abstract
When a language model repairs a wrong reasoning step, the attempt is scored by whether the final answer becomes correct. We argue that this number cannot tell whether the step was repaired, because mending a step and reaching a correct answer are separate events: a model can reach the answer without fixing the step it was told to fix, and it can fix that step and still miss the answer. We introduce a measurement setting that scores the two apart. An external verifier first corrects the generator's arithmetic, which removes every failure a single fixed line would resolve and leaves a residual of genuine reasoning errors. Each remaining failure is anchored to its first wrong step, and the generator rewrites from there. Two independent verifiers then score the rewrite, one on whether the final answer is correct, the other on whether the flagged step still carries its diagnosed error, an outcome we validate against an independent judge and against constructed repairs. Across eight open models on MATH and PHYSICS, we find that the two outcomes come apart in both directions, and that how far they come apart varies more across benchmarks than across feedback conditions. Under a fixed repair signal the step is mended at similar rates on both benchmarks, yet a mended step yields a correct answer for 36.7 to 84.4\% of MATH cases and only 9.2 to 18.3\% of PHYSICS cases. One accuracy number therefore reports two different quantities, overstating repair on one benchmark and hiding it on the other. Varying the feedback along a four-rung ladder from bare location to a procedural hint, we show that repair improves when the feedback teaches a method, not when it names the error: the error-type label moves neither outcome stably despite the diagnosis cost it incurs, while the procedural hint raises both on every model. For a repair loop billed per external-verifier call, this locates the spend that pays and the spend that does not.