Artidoro Pagnoni: "Tokenization as Resource Allocation"

Tokenization is usually treated as preprocessing. This talk argues it is better understood as a resource allocation policy: the choice of input unit, from single bytes to long patches, determines how computation, memory, and data are distributed. Recent work has shown this allocation need not be uniform or fixed, whether through entropy-based byte patching, learned hierarchies, or coarser subword schemes. What it controls depends on the objective. In training, the compression rate is a scaling-law variable. At inference, it sets how often parameters are streamed per unit of output, a binding constraint for current hardware. In distillation, it need not match between teacher and student, and finer-grained students are more data-efficient.