Mechanistic Specialization Does Not Guarantee Performance: Why GPT-2 Outperforms Dual Attention Transformers on Identity Rule Tasks

Dual Attention Transformers (DATs) extend decoder-only Transformers with a dedicated relational-attention stream, making them a natural architecture for abstract identity rules such as ABA and ABB. Surprisingly, we find that comparably sized GPT-2 models outperform DATs on these tasks. We investigate this gap with two complementary mechanistic analyses. First, causal mediation analysis shows that DATs exhibit stronger evidence of hypothesized symbolic mechanisms: symbol abstraction, symbol induction, and retrieval, than GPT-2. Second, a routing analysis shows why this specialization does not translate into better behavior: DATs make more wrong-copy errors, can attend to the correct source token while still predicting the wrong token, and show weak direct contribution from relational attention to the correct-versus-wrong output margin. Ablating positive-routing heads hurts performance, while amplifying those heads improves DAT more than matched controls. These results show that explicit relational inductive biases can shape internal organization without guaranteeing task success.