PhantomFill: When the Form Demands an Answer, Language Models Invent One
Abstract
Language models in production do not write prose. They fill forms: JSON fields, function arguments, extraction templates. We show that the form itself causes hallucination. We ask thirteen models the same question about the same input and change only the answer format across three rungs of an abstention-affordance ladder: free prose, a JSON schema with an explicit escape value, and a JSON schema with required fields. The inputs are constructed so the queried fields are unanswerable by construction, which makes the headline scoring deterministic: no LLM judge decides what counts as fabrication. In free text, GPT-5.5 correctly reports that the evidence is missing in 98% of trials; under a required-field schema, the same model fabricates in 100% of trials (40 of 40). Required fields drive fabrication to 100% in ten of thirteen models. An explicit "insufficient evidence" option rescues only the frontier: all nine open-weight models ignore it. An anti-fabrication system instruction is overridden by the schema in four of six models. Within a single model family, the smallest model refuses impossible schemas, the mid-sized model fabricates at 90%, and the largest emits a machine-readable refusal object of its own design, indicating that resistance to format pressure is a training outcome rather than a scale effect, and one that no current benchmark measures. We release PhantomFill, a contamination-resistant benchmark with two reportable metrics: the Coerced Fabrication Rate and the Escape Utilization Rate.