Prune, Then Select: Select High-Quality, Important, and Diverse Data Using Training Trajectories

The rapid expansion of instruction datasets not only escalates the computational cost of instruction fine-tuning but also brings data-related challenges, such as the presence of noisy or low-quality samples and the redundancy caused by duplicate or highly similar instances. To address these issues, data selection methods aim to reduce training expenses while preserving, or even enhancing, model performance through fine-tuning on an appropriately chosen subset. In this paper, we propose a new method named PS, containing a \underline{\textbf{P}}rune step and a \underline{\textbf{S}}elect step, to ensure selecting a high-quality, important, and diverse subset by efficiently utilizing the training trajectories of data samples collected from a small proxy model. Specifically, in the \underline{\textbf{P}}rune step, we prune low-quality samples that do not exhibit a downward trend in their \textbf{loss trajectories}, as such samples may hinder model training. In the \underline{\textbf{S}}elect step, we introduce \textbf{the learning trajectory}—represented by either loss reduction trajectories or loss reduction rate trajectories—as sample representations for clustering the remaining data. A balanced selection is then performed across all clusters within a fixed budget. We validate PS on the MathInstruct dataset (262K) with the open-source model suite Pythia by comparing it against two categories of data selection methods: importance-based and diversity-based methods. Experimental results show that our PS consistently outperforms all baseline methods across budget constraints of 30K (11.5\%), 50K (19.1\%), and 100K (38.2\%). Notably, PS achieves superior performance with less than 40\% of the data compared to the model trained on the full dataset.