Knowing vs. Saying: A Calibrated Dissociation Test for LLM Agents

LLM-based agents are evaluated almost exclusively by outcome: success rate, error taxonomy, or post-hoc behavioural ablation. None of these can distinguish three mechanistically distinct failure modes: (i) a \emph{world-model failure}, in which the model's internal representation no longer carries the task state; (ii) a \emph{deployment failure}, in which the representation is intact but the generation does not use it; and (iii) a \emph{readout-bottleneck} failure, in which the model's autoregressive policy extracts information that no single-shot last-token probe can recover. We introduce the \textbf{dissociation test}, a per-timestep diagnostic with three components: a linear-probe fidelity $F(t)$, a behavioural accuracy $B(t)$, and the signed gap $G(t) = F(t) - B(t)$. A permutation-derived threshold $\tau$ turns the gap into a calibrated hypothesis test. To make $F$ honest we pair every linear probe with an MLP probe (non-linear upper bound on representational content) and a random-label probe (chance baseline), and we report the resulting \emph{selectivity}. We prove a Fano lower bound linking probe accuracy to recoverable state information, and a \emph{class-relative} gap bound $\mathbb{E}[B]\leq\mathbb{E}[F^\star_{\mathcal{G}}] - \mathbb{E}[D_{\mathrm{TV}}]$ which shows that the \emph{sign} of $G$ is a statement about the probe class $\mathcal{G}$, not about the model. Empirically, on \textsc{Cart-State} (a fully open synthetic environment with exact ground truth, $n\!=\!200$ trajectories, four-rung complexity ladder) and \emph{three architecture families} (GPT-2, Pythia-410M, and the instruction-tuned Qwen2.5-0.5B-Instruct evaluated through its chat template), three findings stand out. First, a naive trivial baseline (default counts/flags, total $0$) scores $0.75$ when graded only on variables with a natural default, above that same slice of every model's behaviour --- but prices have no natural default and are $\sim\!35\,\%$ of all scored cells, so grading the baseline on the \emph{same} full variable set the models are scored on (best constant price guess) gives a matched-scope floor of only $0.49$, which every model clears ($0.65$, $0.65$, $0.61$). We report both numbers because a mismatched-scope floor is itself an easy-to-miss artefact on any mixed discrete/continuous state space. Second, the signed gap $\mathbb{E}[G]\!\approx\!-0.23$ is negative for all three families and all four probe classes ($12/12$ negative; last-token linear/MLP, mean-pooled, attention-pooled), survives a stratified null, and \emph{survives removing all default-valued cells} ($F\!=\!0.24$ vs.\ $B\!=\!0.41$): the model's generation-time read-out beats every static probe, and enlarging the probe does \emph{not} close the gap, so the state is recovered by decoding-time computation, not stored at a recoverable site. Third, a state-matched activation-patching control is directionally consistent on GPT-2 but does not replicate on Qwen; we report it as suggestive rather than conclusive. A base-rate floor and a stratified permutation null rule out the obvious artefacts. All data, caches, and scripts are released under MIT license.