Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Models

Diffusion large language models (dLLMs) generate text through iterative masked denoising rather than left-to-right decoding, creating new challenges for reinforcement learning with verifiable rewards (RLVR). Existing GRPO-style RLVR methods for dLLMs assign a single terminal reward to a sequence of correlated denoising decisions, which can repeatedly reinforce the same trajectory and prematurely sharpen the policy. While this may improve pass@$1$, it can reduce answer coverage and harm pass@$k$ performance. We propose Step-level Risk-Sensitive GRPO (StepRS-GRPO), a simple modification of the policy-gradient advantage for dLLM RLVR. StepRS-GRPO replaces the centered group-relative advantage with a denoising-step-dependent entropic-risk advantage, controlled by a risk coefficient $\beta(t)$. The coefficient is chosen adaptively according to the model’s task-level solve rate and decayed across reverse denoising steps, encouraging stronger reward seeking when coarse reasoning trajectories are formed while avoiding excessive late-stage sharpening during local refinement. Across multiple dLLM backbone scales and reasoning benchmarks, StepRS-GRPO improves the accuracy-diversity tradeoff, achieving competitive or improved pass@$1$ while preserving stronger answer diversity and pass@$k$ coverage.