SpecAgentBench: A Benchmark for Unified Evaluation of Speculative Agentic Systems

Speculative agent systems reduce latency by pre-executing a fast drafter's proposed tool calls and committing each cached result only when a verifier independently selects a matching action. However, existing evaluations are difficult to trust or compare: they omit run-to-run variance, treat agreement with a fallible verifier as task correctness, and report strategy-specific metrics that obscure whether speculation pays. These failures reflect distinct dimensions of evaluation: speculation can be stable yet incorrect, correct yet rejected by the verifier, or frequently accepted yet too costly to help. To evaluate these dimensions, we propose SpecAgentBench on 4,059 tool-calling decisions from four agent benchmarks: it characterizes speed across seeded runs and verification-cost regimes, grades drafts against both verifier actions and task references with partial credit, and places breadth, depth, and caching on a common frontier of committed work versus speculative cost. Across reproductions and controlled evaluations, SpecAgentBench reveals what prior metrics miss: acceptance varies by up to $2\times$ across runs, reported accuracy shifts by up to $3\times$ across model configurations, and apparent speedups can disappear once speculation cost is counted.