ScreenTutor: Learning Computer-Use Agents from Unlabeled Tutorial Videos

Despite the growing promise of computer-use agents (CUAs), their development remains bottlenecked by the lack of diverse, real-world, large-scale demonstrations for training. Human annotation is prohibitively expensive to scale, while synthetic data often fails to capture the complexity and long-tail diversity of real user interactions. We present ScreenTutor, a scalable end-to-end pipeline that automatically turns publicly available, in-the-wild instructional videos into high-quality agent trajectories without human intervention. ScreenTutor searches and preprocesses online videos at scale, filters out irrelevant segments, detects transitions between stable screen states, and identifies the actions performed between them by combining video context, narration, and inverse-dynamics reasoning with vision-language models. Using this pipeline, we introduce ScreenTutorNet, a large-scale dataset of 14.2K multimodal trajectories and over 530K action steps across Windows, macOS, Linux, and thousands of real-world applications. Experiments show that ScreenTutor outperforms recent automated video-to-trajectory baselines on scene detection and action identification, and that mid-training on ScreenTutorNet improves downstream task success on OSWorld and Windows Agent Arena. Together, these results establish ScreenTutor as a scalable approach to converting in-the-wild instructional videos into real-world supervision for more capable CUAs.