Phantom Measurement Artifacts in Multi-Step LLM Evaluation: How a Prompt Bug Inflates Determinism Decay by 14–42×

Accepted paper — no in-person presentation.

This paper remains accepted. No in-person poster presentation is scheduled.

A prompt-construction error in multi-step LLM evaluation — omitting the problem statement from later chain steps — inflates measured determinism decay by 14–42× (95% CIs: 4–60×) and produces phantom measurement phenomena (non-monotone rebounds, rank reversals, apparent cross-family divergence) that are statistically significant and indistinguishable from genuine model behaviour. We measure position-wise determinism D across five frontier models from three vendors (Anthropic, OpenAI, Google) on GSM8K math chains at depths k ∈ {1,3,5,7} under two prompt constructions: a degenerate construction where later steps lose the problem statement, and a fixed construction where every step preserves it. Under the degenerate construction, D decays dramatically (0.98 → 0.24 for Gemini 2.5 Pro) and one model (Claude Haiku 4.5) exhibits a statistically significant non-monotone rebound (+22pp, p < 10⁻¹⁰). Under the fixed construction on the same models and problems, D decays by only 0–3pp — the rebound vanishes, rank reversals disappear, and accuracy at k=7 remains 93–100%. This error was not hypothetical: we discovered it in our own multi-step evaluation pipeline after publishing results based on the degenerate construction. Takeaway: multi-step LLM evaluation is fragile to prompt-construction details invisible in aggregate metrics; a single missing context line transforms a 3pp measurement into a 74pp catastrophe.