When Worlds Collide: Mediating Conflicts Between Parametric and External Knowledge in RAG

Retrieval-Augmented Generation (RAG) augments LLMs with retrieved evidence during inference. During question-answering tasks, retrieved evidence may conflict with an LLM’s internal (parametric) knowledge, leaving models to make silent choices between the two. Meanwhile, users lack the means to steer LLM responses toward the most-preferred knowledge source, even as source credibility varies. To address this, we study LLM inference-time behavior via artificially induced knowledge conflicts where parametric memory is known to directly conflict with external knowledge. Building on established datasets, we construct 7,200 knowledge conflict pairs across 30 Q&A scenarios and four conflict levels (small–extreme). We examine these conflict scenarios across six LLMs and 18 RAG prompts—including the widely used industry defaults—and find that models strongly favor retrieved content, even when it is verifiably false. Paradoxically, this bias tends to increase as the credibility of retrieved content declines. We then evaluate prompt-based user control of model behavior during inference-time conflicts. Two simple prompts—Use Memory and Reject Opinion—increase successful rejection of counterfactual external knowledge from 12.4% to 52.5% and 64.3%, respectively, relative to standard industry prompts. Our findings demonstrate the potential of targeted prompt design for practical, auditable control over LLM reasoning and question answering in knowledge conflict scenarios.