Python is the Counterfactual: Free Per-Child Credit Assignment for Recursive Language Model Agents

Lifelong agents are increasingly built as trees of LLM calls: a parent LLM emits Python in a REPL, and operators like llm_batch() fan out $K$ children in parallel (Zhang et al., 2025). Group-relative RL post-training (Shao et al., 2024) assigns every child the same reward as the parent, conflating sub-skills that mattered with sub-skills that did nothing. We observe that the parent's orchestration logic is Python, not an LLM—and Python is deterministic. We exploit this to compute exact per-child credit by replaying the parent's REPL offline with one child's output masked and measuring the resulting reward drop. Replay is exact, costs $0.037$ ms median per mask (p95 $0.039$ ms, measured on 1500 real episodes), and requires no extra LLM calls. We make four contributions, the first three of which are the headline. (1) A 1500-episode benchmark across three task families (filter, aggregation, paraphrase) constructed so the useful child's output need not appear in the parent's answer; counterfactual replay assigns 0.70–1.00 of its weight to the planted-evidence child at every $K \in \{3,5,8,12,20\}$, while substring-based mention attribution lags by 24.5 points on filter and 33.5 points on paraphrase, and counterfactual is the best-or-tied method in 15 of 15 family-$K$ cells. (2) On $K \le 8$ we compare against exact Shapley (Shapley, 1953) over $2^K$ coalitions and find top-1 agreement of 1.0 and Pearson $r = 1.0$ at $28\times$ less compute by $K = 8$—an equivalence we show holds because these tasks are additive, and which provably breaks under child–child redundancy. (3) We scale $K$ to 100 with oracle children and show counterfactual stays exact at $<26$ ms total per episode while uniform-$1/K$ collapses to 0.01. (4) We package the benchmark as a Prime Intellect environment with a configurable hierarchical_credit flag, induce llm_batch delegation with a single in-context demonstration ($\bar{K} = 8$ on deepseek-chat), and run head-to-head GRPO with Qwen3.5-4B from the same checkpoint. Here our result is negative and we report it as such: delegation collapses under RL pressure—the trained policy abandons llm_batch in favor of direct REPL code, which makes per-child credit structurally inert because there are no children left to credit. A single-seed run on an early task variant (v0.1.1) survived more GRPO steps under hierarchical credit before a zero-advantage-filter crash, but this is confounded with filter survival rather than learning and does not replicate on a redesigned, delegation-resistant variant (v0.2.0). The honest conclusion is that exhibiting an RL benefit from per-child credit requires task complexity that exceeds a single model call's effective single-pass reasoning capacity—a regime our 4B-scale, $\sim 5$ KB-context tasks do not reach, because the model simply does the per-chunk work in its own chain-of-thought. We therefore make the attribution claims, not an RL win, the headline: counterfactual replay is an exact, episode-local, audit-grade per-child attribution primitive for lifelong agents, whose RL payoff awaits delegation-forcing tasks we have not yet built.