On the Promise and Limits of Training LLM Search Agents in Fictional Worlds

Training LLM agents with reinforcement learning (RL) faces a critical bottleneck: environments must be verifiable, support long-horizon trajectories, and scale cheaply. Existing approaches rely on expensive human-curated data or LLM-generated environments. We show that LLMs can instead be trained into capable search agents using rule-generated synthetic environments that are LLM-free, zero-marginal-cost, and infinitely scalable. We build multi-turn RL environments from fictional worlds in which agents must discover documents through search to answer multi-hop questions. Despite sharing no facts with the real world and using only template-generated text, these strikingly simple environments yield agents that transfer to real-world multi-hop search benchmarks, competitive with real-world training environments. Trained agents generalize to unseen fictional universes and learn to scale their search budget roughly linearly with question difficulty, suggesting emergent agentic behavior from environment interaction alone. Probing which design factors drive transfer reveals that hop count matters far more than added constraints or comparisons: even the simplest rule-generated environments are a surprisingly effective, free resource for training generalizable LLM agents.