Arbitrage: Efficient Reasoning via Advantage-Aware Speculation

Modern Large Language Models achieve impressive reasoning capabilities with long Chain of Thoughts, but they incur substantial computational cost during inference. Among these techniques, Speculative Decoding accelerates generation with a fast draft model verified by a stronger target model, yet token-level verification performs poorly on reasoning because small token mismatches can reject semantically equivalent steps. However, due to unnecessary rejections caused by token mismatches in semantically equivalent steps, traditional token-level Speculative Decoding struggles in reasoning tasks. Although recent works have shifted to step-level semantic verification, which improve efficiency by accepting or rejecting entire reasoning steps, existing step-level methods still regenerate many rejected steps with little improvement, wasting valuable target compute. To address this challenge, we propose Arbitrage, a novel step-level speculative generation framework that routes generation dynamically based on the relative advantage between draft and target models. By employing a lightweight router to predict when a target model will produce a meaningfully better step, Arbitrage avoids redundant computations inherent in absolute-threshold baselines. Across different reasoning benchmarks, Arbitrage consistently surpasses prior step-level SD baselines, reducing inference latency by up to ~2x at matched accuracy.