Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization

When a reward is correct on every training example yet consistent with more than one goal, a model can acquire an unintended one, a failure known as goal misgeneralization. Endpoint accuracy on the training distribution cannot tell the two apart, because solving the task and exploiting a surface feature can satisfy the reward equally well. We treat this as a measurement problem: what does a benchmark score measure once a model has been optimized against a signal that is correct but confounded? In a controlled setting, we train language models with GRPO on multiple-choice math problems where the correct answer is always option A, then evaluate on an unseen test set with unbiased correct option positions. Across the families of Qwen2.5, Llama 3.x and Gemma-3 models, biased training often drives option-A rates above $0.90$ in smaller models and collapses unbiased accuracy toward chance, so accuracy stops measuring math ability and instead measures an answer-position policy. We further find reasoning-answer decoupling: capable models continue to generate reasoning that reaches the correct numeric answer while still selecting A. We track this with both numeric extraction from reasoning traces and an LLM judge (GPT-4.1-mini judge, Qwen2.5-3B decoupling rate $\approx 0.66$). Selection no longer follows from reasoning, and accuracy, which is based only on the selected option, scores both channels as a single number. The broken construct does not stay in domain: biased models inflate A-rates on out-of-domain MMLU and value-laden prompts. Continued training on unbiased data reverses the in-domain shift unevenly and only partially reverses the out-of-domain one, so a model can look restored on its training distribution while remaining biased on inputs it never saw. We treat reasoning-answer decoupling rate, the fraction of items where reasoning reaches the correct answer while selection does not, as a measure that endpoint accuracy cannot recover; together with the answer distribution and out-of-domain behavior, it separates capability loss from a learned, transferable shortcut.