Reasoning Ability Is Not Reasoning Economics: Benchmarking Sequential Budget Allocation in LLMs
Yashaswi Sharma ⋅ Prabhu Pugalenthi ⋅ Muyan Weng ⋅ Defu Cao ⋅ Yan Liu
Abstract
Large reasoning models are typically evaluated on what they can solve, not on what solving costs. We introduce ReasoningEconomicsEnv, a multi-turn benchmark and reinforcement-learning environment in which a model must answer a sequence of ten math problems under a single shared token budget that is observable in the state and depleted by every token the model emits. Evaluating 14 frozen open-weight models (7B to 72B; spanning instruction-tuned, math-specialized, reasoning-distilled, and native-thinking-mode models; dense and MoE architectures) over 50 seed-matched episodes each, we find that task accuracy and budget economics dissociate almost completely: accuracy spans only $0.228$ to $0.302$ (bootstrap 95\% CIs mostly overlapping), while episode reward spans 40.7 points ($1.18$ to $-39.50$), with reward rank-ordered by token spend (Spearman $\rho=-0.996$) rather than by correctness. Thinking-mode models spend 2.4 to 4.5 times as many tokens as non-thinking models, while accuracy actually decreases only slightly, by 0.041 (95\% CI $[-0.056, -0.026]$), in this constrained-budget regime. Per-turn spend profiles are flat in turn index for every model class: models continue spending at an unchanged rate long after the budget is exhausted, indicating that frozen LLMs are budget-blind even when the remaining budget is stated in the prompt. The environment exposes a verifiable, decomposable reward (correctness, cost, overspend, efficiency, terminal utilization bonus) and is directly usable as a substrate for Reinforcement Learning from Verifiable Rewards (RLVR)-style post-training. We will release the environment, evaluation harness, and all trajectories upon publication.
Chat is not available.
Successful Page Load