CLIO-BENCH: Beyond Accuracy in AI Historical Judgment

Foundation models increasingly answer questions whose correctness is either counterintuitive or not externally checkable. In such settings, final-answer accuracy is an incomplete scientific object: a model can recall the right answer with little investigation, hedge away a counter-prior truth, or present its own stable disposition as evidence. We introduce Clio-Bench, a 47-claim historical-judgment benchmark designed to separate measured correctness from groundedness: 38 resolvable claims with coded truth, including counterintuitive cases where the popular answer is wrong, and 9 contested claims with no gold label. We evaluate 16 model backends, including MiniMax-M3, Qwen3.7-Plus, Kimi-K2.6, and GLM-5.2, under a single-agent search protocol and TruthGPT, an evidence-grounded debate protocol with per-persona model routing. Across more than 1,000 panels, the core result is not that debate reliably improves accuracy. Rather, debate is a probe and design principle for groundedness and depth: useful panels read more, preserve opposition, challenge specific evidence, and expose model temperaments. Accuracy gains on counterintuitive claims are a diagnostic byproduct; evidence-starved or persona-collapsed debate can reduce accuracy. These measurements identify an open post-training problem: learning models to debate in the grounded sense, not merely to produce persuasive multi-agent transcripts.