Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization

Sycophantic agreement refers to a behavior in which language models excessively affirm the user, often at the cost of factual accuracy. Although sycophantic agreement is a well-known failure of model alignment, there is limited understanding of how it emerges from model training. In this work, we demonstrate that sycophantic agreement can emerge as an unintended consequence of widely used contrastive preference optimization objectives. Using the OLMo 3 DPO pipeline, we show that for various pairs of teacher models across three families, there is a strong correlation between the log-ratio of the teacher models' sycophantic agreement rate and the resulting student model's sycophantic agreement rate. The sycophancy signal is diffused across the entire dataset rather than concentrated in a sparse set of examples: there are no explicit instances of sycophantic agreement, and data attribution-based filtering methods fail to mitigate sycophancy without removing a large portion of the dataset. Furthermore, we demonstrate that this unintended transfer from teacher models to students is not limited to DPO but can also occur across 5 prevalent preference optimization objectives. Based on our findings, we propose a lightweight mitigation that teaches a model to maintain its answer. Overall, our findings suggest that the teacher models used to generate preference data can interact with alignment training objectives in unexpected ways, generalizing to undesirable and potentially harmful behaviors like sycophantic agreement.