Replayable Trajectory Dataflow: Auditing Reward Measurement under Optimization and Drift in RLVR

In reinforcement learning with verifiable rewards (RLVR), the verifier *is* the measurement instrument: it scores every sampled trajectory, and those scores steer the policy toward higher-measured behavior. This makes the verifier both a target of strategic optimization—reward hacking, where the policy games an exploitable proxy—and a source of non-stationarity, when its version silently drifts mid-run. Both are measurement failures, yet each surfaces only after the contaminated trajectories have already shaped checkpoints, and stochastic generation prevents faithful reproduction by rerunning. We propose Replayable Trajectory Dataflow (RTD), a lineage abstraction that materializes each trajectory as an immutable record and re-measures it under a corrected reference without re-sampling. RTD partitions run state by recomputability, then answers three queries over the store—rescore, diff, and localize—to detect a gamed or drifted verifier, quantify how far the true objective diverged from the recorded reward, and pinpoint the first update the failure corrupted. On an RLVR prototype, RTD surfaces a hidden $0.89 \to 0.05$ accuracy collapse invisible to the rising training reward, localizes a verifier regression exactly (100% disagreement, three seeds), and guides rollback-based recovery at a fraction of restart cost, while re-sampling reproduces only 2% of the historical outputs. RTD reframes post-training evaluation failures as cheap, auditable queries over materialized measurement evidence.