Readable but Not Controllable: Neuron-Level Evidence for Medical LLM Hallucination

Hallucinations are a central obstacle to deploying medical language models, yet it remains unclear whether the internal representations that signal hallucination can be used to control it, rather than merely detect it. Using four open-source models on a suite of medical question-answering datasets, we show that linear probes on intermediate MLP activations detect hallucination with AUROC between 0.77 and 0.86. Examining how this signal is organized, we find it is distributed and redundant rather than localized: the most individually discriminative neurons outperform random neurons only at very small subset sizes, random subsets of a few hundred neurons recover nearly the full signal, sparse probes retain thousands of neurons, and low-dimensional random projections preserve most of the detection performance. We then test whether this representation is causally actionable. Across 16 model--dataset combinations, steering hallucination-associated neurons does not reliably outperform steering random neurons after multiple-comparison correction, and detection strength does not predict intervention effect. Medical hallucination is thus readily decodable from internal activations but resistant to neuron-level control, a separation between what a representation reveals and what it lets us change.