Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining

Measuring the influence of training data on final model performance consistently across pretraining is challenging due to the difficulty of selecting evaluation tasks representative of general model capabilities, as well as the dependence on task performance at intermediate checkpoints. To enable a task-agnostic measure of training data influence, we reformulate influence as a measure of how individual data points shift the model toward its final parameters. Specifically, we define an example's influence by how much its gradient update reduces the distance to the final model weights, and estimate this quantity from intermediate checkpoints without retraining. Applying this method to the full training trajectory of two open-weight LLMs, we find that the influence of different data domains evolves systematically over time: notably, literature and STEM data exhibit a crossover in influence, with literature dominating early training and STEM becoming more influential in the latter half. These findings hold across experimental settings and suggest that our task-agnostic influence measure offers principled insights into how training data shapes the capabilities acquired during pretraining, with implications for data composition and curriculum design.