Are Non-BPE Tokenizers Ready to Improve LLM Numeracy?

Arithmetic remains a persistent weakness for large language models (LLMs). However, major research efforts have primarily focused on tool use or reasoning chains, which bypass this foundational limitation rather than addressing it directly. While tokenization has emerged as a promising line of research to tackle this numeracy issue, it still lacks a systematic evaluation. Regarding this gap, our paper asks whether alternatives to standard Byte-Pair Encoding (BPE) tokenization can improve arithmetic while preserving general natural-language (NL) ability. We study three tokenizer innovations over a (fixed-vocabulary) BPE baseline: 1) \textit{cross-tokenizer distillation} for controlled digit chunking, 2) \textit{specialized number encodings} that represent numerical text with a \verb|| token, and 3) \textit{byte-level LLMs} that operate on bytes instead of token-level. We build an evaluation protocol that requires LLM arithmetic capabilities to solve problems in both explicit and implicit ways. Surprisingly, our result shows that all three current alternatives could not outperform BPE in the arithmetic-NL Pareto frontier.