Verify to Intervene, Not to Score: Baseline-Relative Verification for Small-Model Self-Consistency
Jai Kumar Sharma ⋅ Peeyush Tapadiya
Abstract
External verifiers (process reward models, PRMs) are usually evaluated as *absolute* trace scorers, summarized by a pooled AUROC. But a small-model system using self-consistency *already has an answer*, the agreement plurality, so the deployment question is baseline-*relative*: should an imperfect verifier *override* that answer? We show pooled trajectory AUROC is provably insufficient for this decision: a per-question score shift leaves every within-question ranking and score-difference decision unchanged while moving pooled AUROC substantially ($0.94 \to 0.56$), so two verifiers of near-identical AUROC can have very different deployment value. Using the *same* PRM scores as a majority-relative *cluster margin*, scoring only the few cluster representatives and overriding when a non-plurality cluster outscores the plurality, gives statistically significant net gains on the primary $\le 3$B tuples ($+0.127$ on GSM8K/SmolLM2/Skywork, $+0.063$ on MATH/Qwen/Skywork), whereas *absolute* thresholding of the same scores gives substantially smaller gains. At matched total compute the override beats spending verifier FLOPs on extra self-consistency where minority-correct opportunity exists, and ties on strong-majority tuples. The practical message: at small scale, deploy off-the-shelf verifiers as within-question, harm-budgeted comparators against the consensus answer, not as pooled scorers.
Chat is not available.
Successful Page Load