Is a Plan Load-Bearing? A Black-Box Causal Audit of Plan-Based Oversight for LLM Agents

Plan and CoT-based monitoring of LLM agents assumes that the plan an agent states is behaviorally meaningful, i.e., that reading the plan tells an overseer which irreversible actions the model will take. This assumption has been highlighted as unvalidated by recent monitorability agendas, and no black-box experimental design tests it over tool-executing trajectories. We introduce a validity-gated counterfactual test: on $\tau^2$-bench tasks rewritten so that a customer is explicitly indifferent between two outcomes (e.g., refund vs. exchange), we verify behaviorally that both outcomes are achievable and policy-compliant, have the agent ratify a plan for each, then counterfactually seed one or the other as the agent's own stated plan and measure which tools the agent calls. Across four open-weight models on identical inputs, the seeded plan causally shifts the outcome in every model, but to widely varying degrees. Binding ranges from $0.88$ (Qwen3-30B: $P(\text{outcome A} \mid \text{plan A}) = 0.94$ vs. $P(\text{outcome A} \mid \text{plan B}) = 0.17$) down to $0.58$ (GLM-4.5-Air). The failures are not noise, but instead correlate with the model's revealed preference, measured from a handful of unseeded rollouts. In all four models, overrides occur when the seeded plan conflicts with this preference ($2\% \to 22\%$, $5\% \to 39\%$, $10\% \to 75\%$, and $6.5\% \to 48\%$ agree-to-conflict; held-out preference splits reproduce the contrast in every model; Qwen task-level bootstrap difference $0.19$, 95% CI $[0.08, 0.31]$). Overrides are silent, uncorrelated with the model's own stated validity judgments, and highly model-dependent. Two models carry near-global preferences that mirror each other on the same tasks, the two GLM-family models' preferences are strong but task-specific, and between 3 and 19 of 59 tasks per model are fully seed-invariant, with the realized outcome matching the measured preference in 40 of 41 cases. These results support plan-based oversight as conditionally valid, and yield a practical black-box audit: measure an agent's preferences with a few unseeded rollouts, and do not trust stated plans that conflict with them.