Kalshi Bench: Outcome-Grounded Calibration as a Contamination-Resistant, Non-Stationary Measurement Primitive

Most LLM evaluations are \emph{reference-matching}: they score output against an authored key --- a gold translation, a multiple-choice answer, a human judge's preference. Viewed as measurement instruments, reference-matched benchmarks share three pathologies that the AI-measurement-science agenda foregrounds. They are \emph{gamed and contaminated} once their items are public; they are \emph{frozen} against a world whose concepts and labels drift; and a single scalar \emph{conflates distinct constructs}. We argue that \textbf{outcome-grounded calibration} --- scoring a model's stated probability against an external event that did or did not occur --- is a measurement primitive that addresses all three \emph{structurally} rather than procedurally. Kalshi Bench is a Verifiers-compatible environment that scores models on real, already-resolved Kalshi prediction-market events with the Brier score: a strictly proper scoring rule whose properness is exactly the incentive-compatibility property an evaluation needs to resist confidence gaming; whose item pool refreshes as new markets resolve, giving a live test surface that resists the freeze-and-leak failure of static benchmarks; and whose Murphy decomposition separates calibration from discrimination, recovering two constructs that a scalar leaderboard collapses. We release a single-file environment with explicit data-leak controls, an end-to-end harness wired through the Prime Intellect platform, and a stratified evaluation of six frontier models (GPT-5, GPT-5.1, Claude Opus 4.7, Claude Sonnet 4.5, Gemini 2.5 Pro, DeepSeek V3.2-exp) across five Kalshi categories ($n=600$ model-market evaluations). Four findings of measurement-methodological interest: (i) the leaderboard is non-monotone in model scale --- DeepSeek V3.2 leads at Brier 0.874, GPT-5/GPT-5.1/Claude Opus tie at $\approx 0.803$, Claude Sonnet 4.5 trails at 0.766, and Gemini 2.5 Pro falls below \textsc{Uniform} (0.602); (ii) a Murphy decomposition shows DeepSeek's dominance is driven by \emph{resolution} (sharp discrimination), not calibration, while GPT-5 is the best-calibrated frontier model but only middle of the pack on overall Brier --- the best-overall and best-calibrated model are different, so a scalar leaderboard is an invalid summary of two separable constructs; (iii) the per-category picture is uneven, with Mentions sitting below \textsc{Uniform} across all six models; (iv) the leak controls hold at scale (0/600 forbidden tokens), and the extraction reliability of thinking-mode models is itself a measurable audit dimension --- Gemini 2.5 Pro lost 20\% of its responses to incomplete output, a measurement artifact of the agent's generative process rather than of its beliefs. We use these results to sketch a measurement-science research program across the workshop's three axes --- interactive, strategic-optimization-robust, and non-stationary measurement --- and release the analysis script that reproduces every number.