TokEval: A Tokenizer Analysis Suite

Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream performance. We introduce TokEval, a framework of tokenizer evaluation metrics that goes beyond standard measures like fertility and compression rate to capture linguistically and structurally meaningful properties, e.g., UTF-8 character boundary integrity and digit place-value boundary alignment for mathematics. To validate whether these metrics are predictive of downstream model performance, we conduct controlled language model pretraining experiments across tokenizers varying in training data mixture, pretokenization strategy, and training algorithm (BPE, SuperBPE, UnigramLM). We evaluate the resulting models on bits-per-byte and several targeted benchmarks, spanning linguistic understanding, mathematical reasoning, and code generation. Our experiments reveal that some intrinsic metrics correlate strongly with specific downstream abilities. These findings suggest that principled intrinsic evaluation can substantially reduce the cost of model development by identifying promising configurations before committing to expensive pretraining runs.