TAPIR: A Neural Pointer-Based Parallel Tape Machine

We present a family of architectures we dub \textbf{TAPIR}: A Pointer-Based Parallel Tape Machine, in which read and write heads are maintained as pointer distributions over a differentiable tape. The pointers move along the tape through a convolutional operator. This lets us decouple the \emph{size} of the tape (memory) from the length of the sequence, traditionally a restriction with transformers. The pointer-based approach provides favorable inductive biases such as being able to copy spans of text, which lead to improved performance on a suite of synthetic tasks.