Sober Metrics: Output-Token Efficiency Scores Disagree, and Their Rankings Are Statistically Fragile

Scalar efficiency scores such as ACU (accuracy per kilotoken), cost-of-pass, and token-AUC now rank reasoning systems on efficiency leaderboards. Whether these scores agree with one another, and whether the rankings they produce are statistically resolved, has gone untested. We study them as reduced functions of accuracy and mean output tokens, so every claim here is about output-token efficiency, not deployment cost, and we measure their agreement and bootstrap resolution across a grid of real reasoning systems on a common benchmark. A single worst-case rank correlation $\tau_{\min}$ conflates three tiers of disagreement: the alarming number is the definitional accuracy-versus-cost axis, whereas genuine disagreement between composite metrics is far milder, yet these metrics still crown different, non-overlapping champions. The rankings are also weakly resolved: the accuracy winner is the unique top system in only 47.8% of problem resamples, and in the (possibly tied) top set in 79.0%. A companion theorem explains the structure: all monotone, unit-invariant, Pareto-consistent scalars agree exactly on the Pareto partial order, so every scalar is one arbitrary completion of it. Efficiency comparisons should therefore report metric-choice sensitivity, bootstrap resolution, and a Pareto band, not a single scalar.