Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks

Existing web agent benchmarks have largely converged on short, single-site tasks that frontier models are approaching saturation on. However, real-world web use consists of long-horizon, multi-site workflows. Common web navigation tasks such as comparing products across different domains, planning trips across multiple services, or summarizing information from multiple search queries, require sustained context, cross-site reasoning, and coherent planning over potentially hours of browsing. To capture and evaluate such behaviors, we introduce Odysseys: a benchmark of 120 long-horizon web tasks derived from real world browsing sessions, evaluated on the live Internet. We find that binary pass/fail evaluation is inadequate for long-horizon settings, and introduce a rubric-based evaluation, annotating each Odysseys task with an average of 5.8 graded rubrics. We demonstrate that this yields higher agreement with humans, and provides a more fine-grained signal than commonly used trajectory-level LLM-as-a-judge evaluation metrics. We test several leading frontier models, and find that the strongest models achieve a success rate of 53%, which leaves substantial headroom for future improvements. Odysseys isolates the critical evaluation of long-horizon proficiency in open-web environments, providing a realistic benchmark to measure progress towards computer-use agents that can potentially productively operate for hours.