Auditing Efficient Reasoning Routers Under Distribution Shift
Abstract
Adaptive test-time compute promises efficient reasoning by routing each query to the cheapest useful strategy: direct answering, chain-of-thought, planning, tool-like reasoning, code execution, or multi-strategy fallback. We ask a deployment-facing question: when such routers improve average accuracy-cost frontiers, what reliability failures remain hidden under distribution shift? Using a repaired Select-then-Solve/PARADIGM outcome matrix, we audit 3,040 leave-one-domain-out routing decisions across 10 benchmarks and four model families. Cost-aware PRISM-like routers reduce top-1 errors but still leave 12.66--14.80 percentage points of residual route regret. Semantic-KNN, RTR-style, and adapted public RTR-code collision controls do not remove the gap. On exact GPQA Diamond evaluations, STS-trained routers under-route to Direct and leave a 19.70 point oracle gap; optimistic selected-set scoring overstates deployable fallback because majority aggregation leaves 13.13--14.14 points of regret for the strongest fallback slices. Finally, a simple repair probe---run Direct and CoT first, then escalate only when they disagree---cuts GPQA deployable regret from 19.70 to 9.60 points while spending 3.4x the router's token cost. The result is an audit protocol and a concrete reliability-cost trade-off for efficient reasoning systems.