SODA: Semi On-Policy Black-Box Distillation for Large Language Models

Black-box knowledge distillation for large language models presents a strict trade-off. Simple off-policy methods (e.g., sequence-level knowledge distillation) struggle to correct a student's inherent errors. Fully on-policy methods (e.g., Generative Adversarial Distillation) solve this via adversarial training but introduce well-known instability and crippling computational overhead. To address this dilemma, we propose SODA (Semi On-policy Distillation with Alignment), a highly efficient alternative that leverages the inherent capability gap between frontier teacher models and much smaller students. Since a compact student model's natural, zero-shot responses are almost strictly inferior to a powerful teacher's outputs, we construct a highly effective contrastive signal simply by pairing the teacher's responses with a raw, one-time static snapshot of the student's rollouts. By seamlessly translating this gap into a concise preference optimization pipeline, we expose the student to its own static inferior modes. This achieves high-quality distribution alignment while completely eliminating the need for complex filtering, costly dynamic rollouts, or fragile adversarial training. Extensive evaluations across four compact Qwen2.5 and Llama-3 models validate this semi on-policy paradigm. SODA matches or outperforms state-of-the-art methods on 15 out of 16 benchmark settings. More importantly, it achieves this comparable distillation quality while training 10x faster, consuming 27% less peak GPU memory, and entirely bypassing adversarial instability.