Can RLVR Recover Reasoning Quality Missing from Mid-Training?

The standard reasoning-model recipe distills chains of thought (CoT) by supervised fine-tuning (SFT) as a mid-training stage, then applies reinforcement learning from a verifiable reward (RLVR). Curating high-quality reasoning traces is costly, whereas RL rollouts consume GPU compute, raising a practical question: can additional RL compensate for low-quality reasoning examples in mid-training? We test this with a 2×2 grid that factors each SFT target into (reasoning quality, answer correctness). Across four base models (1.5B–7B), evaluated in-domain on math (the RL prompt domain), RL with an answer-only reward does not close the good/bad-CoT gap. RL consistently improves models trained on good reasoning paired with wrong answers (the scaffold) but not models trained on correct answers paired with structurally broken reasoning (the deficit); the deficit changes little on answer accuracy and, at roughly constant accuracy, on a process-level reasoning rubric whose judge never sees the reference answer. Raising the deficit's rollout budget ~20× drives its training reward from .16 to .93 with little change in held-out accuracy. A small additional SFT run on 2,000 good examples, at ~1/10th the training compute of one standard RL run, closes 94% of the gap on GSM8K. The scaffold–deficit contrast replicates across three RL seeds on the main base model, and the deficit result in single-seed runs on three further base models. Under the settings evaluated here, additional answer-only RL does not substitute for direct supervision with good reasoning traces.