Reasoning Tokens Buy Competence, Not Compliance: What Test-Time Compute Does and Does Not Purchase, Measured with a Sign-Exact Oracle
Abstract
Test-time reasoning is usually evaluated on one axis: does more thinking buy more accuracy per token? We add a second axis that efficiency work has ignored, compliance: when an explicit instruction opposes a convention the model absorbed in training, do reasoning tokens help the instruction win? Using CircuChain v2, a circuit-analysis probe whose answers are verifiable to the sign by four independent solvers, we measure both axes across 18 model configurations from 7 families. Three findings. (1) Reasoning buys competence only: toggling the same weights between reasoning and non- reasoning modes at five scales (Qwen3 1.7B–32B) raises magnitude accuracy at every scale (from 1.7×at 32B to 12×at 1.7B) while reversion to the trained convention stays within a narrow band: 63–81% (configuration mean 74.2%; pooled over variables 75.9%), a band that also contains GPT-5, Gemma-4, GPT-OSS-120B, and DeepSeek-V3 across a 792×competence range. (2) The GPT-5 reasoning-effort ladder dissociates the axes: at minimal effort it answers from the prior, cheaply and wrongly; at low effort it solves sign-trap circuits to nine decimals yet still reports the trained frame∼71% of the time; at default effort reasoning consumes the entire token budget and zero of 121 requests return an answer, at the highest cost per instance. (3) Unbounded rumination is a real cost: at 4B–32B, reasoning modes burn the full 16k budget on 73–77% of problems without converging (the 1.7B model instead fails fast, truncating on only 16%). Across all models, flipping one convention instruction breaks 401 previously-solved problems and fixes 6. Efficiency evaluations that report accuracy-per-token implicitly assume compute also purchases controllability; on this benchmark, at every budget we tested, it does not. The generator, oracle, and harness will be released; total study cost was under $100