Playing with Fire: What Transfers When RL Trains a Language Agent?

What does reinforcement learning (RL) teach an agentic language model, task-transferable competence or scaffold-specific behavior? We study this question in Hanabi, a cooperative hidden-information card game, by training and evaluating models across three Holmesian scaffolds that present the same game with different prompt support: WATSON supplies the public game state, visible hands, clue-derived own-card information, and legal moves; SHERLOCK adds Hanabi rules and engine-computed deductions over possible card identities; and MYCROFT requires the model to update per-player belief states from intervening actions before acting. Our results reveal that a QWEN3-4B model with weak scaffold-independent Hanabi competence, trained with RL on the supported SHERLOCK and MYCROFT scaffolds, achieves large in-domain gains but fails to transfer to held-out WATSON, where rules are not provided in context. In contrast, GPT-OSS-120B, which starts with stronger scaffold-independent Hanabi competence, improves on both held-out WATSON and SHERLOCK after RL on MYCROFT alone. To evaluate long-horizon state tracking, we construct FIREWORKS, a 10K benchmark requiring exact reconstruction of per-player hidden card beliefs across 2–5 player games, where QWEN3-4B has near-zero Pass@1. Training on FIREWORKS yields QWEN-ST, which improves SHERLOCK performance over QWEN3-4B despite using the same 4B backbone, showing that the 4B model can benefit from RL when the scaffold supplies the missing Hanabi rules and deductions. These results suggest that cross-scaffold transfer is gated by pre-RL task competence: playing Hanabi well without prompt-supplied rules rather than by model scale alone. QWEN-ST also produces longer, more persistent rollouts and improves held-out non-Hanabi PI-LOGIC exact-answer Avg@8/Pass@8 from 0.03/0.15 to 0.13/0.37, with continued Hanabi RL raising them to 0.20/0.61, even though the exact belief-reconstruction skill does not transfer zero-shot to MYCROFT. When MYCROFT state-tracking and move-rating rewards are optimized jointly, cold-start training is slowed by reward interference, while initialization from FIREWORKS reaches the same state-tracking reward level in roughly half as many RL steps. Together, our results separate RL post-training into three outcomes: scaffold-local gains, competence-gated transfer, and computation-style/trainability transfer, where hard-task RL changes rollout behavior and improves downstream optimization even when the trained zero-shot skill does not transfer across scaffolds.