Mode-seeking in on-policy distillation of language models

On-policy distillation is gaining traction for posttraining of large language models due to its ability to teach models new behaviors in a way that is most natural to them. Yet, our understanding of its properties and of when it should be used remains limited. We make a step in that direction by revisiting the classical mode-seeking behavior of reverse KL divergence. Through theory and experiments in a controlled setting, we characterize what mode-seeking looks like in large language models, and we demonstrate that on-policy distillation can deflate entropy compared to the teacher, whereas off-policy methods like supervised fine-tuning provably inflate it.