Compliments to the Complementizer: Indirect Routing of Syntactic Structure in a Transformer Language Model

Unlike recurrent neural networks, which have a fixed path for accumu- lating information from the input, transformer language models’ use of attention accommodates more flexible regimens for how information is routed through the model. We investigate how syntactic information about long-distance filler-gap dependencies is routed in transformers. We focus specifically on German sentences involving ”verb-second” structures. In such sentences, a language model will need to encode information about a fronted element and move it to downstream positions in order to modulate expectations about possible continuations. Through interchange interven- tions on contrastive minimal sentence pairs, we identify that transformers make use of an indirect path routing syntactic information first from the fronted element to the complementizer and from there to the downstream position at which predictions are made. This non-trivial detour of infor- mation through the complementizer aligns with linguistic theories of the structure of long-distance filler-gap dependencies. This information is echoed in abstractly similar dependencies in English and Italian, but not in cases of dependencies that are semantic rather than syntactic in nature. We then explore the content of the model’s internal representations by training a hierarchical Tensor Product Decomposition Network (TPDN) that identifies the model’s representation of grammatical features. We can then reconstruct model activations at the complementizer position under the assumption that this neurosymbolic representation captures the es- sential aspects of the model’s encoding of grammatical structure. Causal interventions using these reconstructions confirm that this manipulation of the grammatical features produces the predicted changes in downstream continuation predictions.