LoRA Scaffolded Policy Optimization (LSPO): A Sampling- Time Low-Rank Scaffold for Recovering Reinforcement- Learning Gradient on Zero-Reward Cliff Prompts

We introduce LoRA-Scaffolded Policy Optimization (LSPO), an extension of GRPO (Shao et al., 2024) for math-reasoning RL that addresses cliff prompts—prompts whose entire group of K rollouts fails, contributing no policy-gradient signal. At each training step, LSPO detects cliffs from the base rollouts, fits a fresh rank-16 LoRA adapter (Hu et al., 2022) on the dataset’s ground-truth cliff solutions for a few SFT steps, re-rolls out the cliff prompts under the resulting base+LoRA policy, and splices any verifier-passing rollouts into the GRPO batch as positive examples; the GRPO step itself runs with the LoRA branch disabled, so the RL gradient lands on the base only. On DeepMath-103K (He et al., 2025) with DeepSeek-R1-Distill-Qwen-1.5B (DeepSeek-AI, 2025) we evaluate LSPO against a matched DAPO (Yu et al., 2025) baseline over n=5 paired seeds and 1000 training steps. LSPO’s 5-seed mean matches or beats DAPO on all 16 (benchmark, pass@k) cells, with improvements of up to +10.7 pts on AIME24 pass@4, +6.7 pts on AIME24 and AIME26 at pass@16, and +2.4 pts on MATH500 pass@1. Detailed paired analysis, cliff-conversion efficacy, and stability characterization are in the appendices. We will do our best to make training artifacts (code, driver logs, and per-step traces) available upon reasonable request, subject to an institutional release review.