Poster
in
Workshop: Second Tokenization Workshop Fri, Oct 9, 2026 • 11:20 AM – 12:30 PM PDT Continental Ballroom 1

Compute Optimal Tokenization

Tomasz Limisiewicz ⋅ Artidoro Pagnoni ⋅ Srini Iyer ⋅ Mike Lewis ⋅ Sachin Mehta ⋅ Alisa Liu ⋅ Margaret Li ⋅ Gargi Ghosh ⋅ Luke Zettlemoyer

Project Page

Abstract

Scaling laws enable the optimal selection of data amount and language model size, yet the impact of the data unit, the token, on this relationship remains underexplored. In this work, we systematically investigate how the information granularity of tokens, controlled by the compression rate (i.e., average bytes of text per token), affects scaling trends. We train 988 latent tokenized models (BLT) from 50M to 7B parameters that enable setting the desired compression rate. This flexibility allows us to study the role of compression rate well beyond 4.57 bytes per token obtained by a popular BPE tokenizer. Our experiments reveal that in compute-optimal configurations, model parameter counts scale proportionally to data size measured in bytes, not in tokens as commonly perceived (Kaplan et al., 2020; Hoffmann et al., 2022). Furthermore, we discover the optimal compression rate is lower than the one obtained with BPE and further decreases with compute. These findings generalize to both latent and subword tokenization, guiding language model developers on tokenization scheme selection for maximal compute efficiency.