Does the Proxy Preserve the Decision? A Reference-Sensitivity Audit of Multiple-Choice Reformulations of Incremental QA
Abstract
Multiple-choice reformulations are widely trusted as cheaper proxies for open-ended tasks, but a proxy is only faithful with respect to a chosen property. One property that matters is timing: not just what a model answers, but when it appears ready to act. We show that the familiar checks used to license such a substitution cannot certify that this decision survives that substitution. We audit this in incremental quizbowl, where questions unfold clue by clue. On 2,258 retained items, the proxy presents itself as clean under familiar checks: answer choices alone add little signal, and prefix-wise probabilities pass coarse calibration screens. Yet a fixed-confidence audit reports no timing difference only because every non-crossing trajectory is forced to the final clue. The reassuring zero is in fact a ceiling artifact that would hide real shifts. Richer continuation analyses do not settle the question; they reveal how much the answer depends on the evaluator. Across five constructed QA references and two calibration choices, the mean estimate ranges from the QA reference stopping 1.63 prefixes earlier to MC stopping 0.50 prefixes earlier. A learned next-state model produces apparent item-level timing differences, but it uses gold similarity and future-prefix features, so it is a clairvoyant stress test, not a deployable estimate. The core implication is that acceptable answer-level performance, low detected leakage, low coarse calibration error, and even a zero estimated stopping shift can all coexist with no valid evidence that a proxy preserves the decision to answer, wait, use a tool, or escalate. We therefore distill these failure modes into a reusable decision-preservation audit card for benchmark translation: a portable, claim-licensing instrument rather than a single result. Because we do not observe paired open-ended probability trajectories, we do not claim that multiple choice changes the true stopping policy. We show why current evidence cannot certify that it preserves one.