Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

Multi-task model merging combines separately trained experts into a single model. Standard practice freezes each expert at its optimal validation loss before merging. We challenge this convention by systematically varying expert training duration across five domains, five merging methods, and two model sizes (Qwen3.5 0.8B and 2B), saving checkpoints from 25% to 500% of the optimal training steps. We find a striking method-dependent pattern: simple averaging degrades with overfitting, while sparsification-based methods like TIES reach their best performance well past the validation optimum. We explain this through a bias-variance decomposition that parallels how random forests benefit from high-variance learners. These results suggest that training duration and merging method should be chosen jointly rather than independently.