Can LLMs Predict the Future? PolyBench: A Benchmark for Future Prediction from Real- World Markets

Can large language models predict the future, and does scaling capacity help? We introduce PolyBench, a benchmark built from the public Polymarket record of resolved binary prediction markets that uses each model's training-data cutoff to separate real prediction (events resolved after the cutoff, with knowable outcomes that the model could not have seen during training) from recall. Our evaluation spans three axes: model (six models in three families with two variants each: Claude Sonnet, Claude Opus, Qwen2.5), time (pre- vs.\ post-cutoff), and event category (nine domains). Three findings emerge: frontier LLMs barely predict the future (the strongest open-weight model fails to beat the empirical base rate, and the four Claude models clear it only marginally); scaling capacity helps only below the frontier (open-source Qwen improves substantially with size, the frontier Claude lineup plateaus across versions and tiers); and event category drives post-cutoff prediction more than within-frontier capacity does.