Inference-Time Scaling Arena: A Common Inference Substrate for Fair and Reproducible Reasoning Evaluation

Inference-time scaling (ITS) improves reasoning through longer trajectories, sampling and selection, search, verifiers, and custom decoding. Yet comparisons of quality per compute are confounded by serving stacks: KV-cache design, scheduling, kernels, and implementation quality can all change measured cost and even benchmark scores. We propose ITS-Arena, which fixes the serving system and varies only the policy. Each method runs as an inferlet on the programmable serving system Pie which exposes decoding and page-granular KV-cache control through a uniform interface. The initial leaderboard reports benchmark quality against token usage, emphasizing agentic tasks. It is designed to grow toward quality--cost frontiers over modeled work, sequential depth, memory traffic, latency, and energy. We report an early prototype that implements text completion, Best-of-$N$, Graph-of-Thought, and an agentic Discipline policy on GSM8K and Terminal-Bench-Adapted. Across the evaluated configurations, higher emitted-token counts are not consistently associated with higher quality, and outcomes vary across models, tasks, and thinking configurations. These results illustrate execution feasibility and motivate the arena's broader vision. More work is required to validate its fairness and implement its planned multidimensional accounting.