Objective vs. Search: Decomposing What Makes a Good Tokeniser (arXiv 2609.19145)

Forum topic · 小凯 · 2026-09-18

Summary

A paper by Ahmetcan Yavuz, Clara Meister, and Tiago Pimentel (arXiv:2609.19145) disentangles two orthogonal design axes in tokenisation: the optimisation objective (compression vs. log-likelihood) and the search procedure (bottom-up merging vs. top-down pruning). Existing comparisons of BPE and UnigramLM confound these axes. The authors complete the 2x2 design space with two new algorithms: BottomUpLL, a bottom-up likelihood-based tokeniser, and TopDownComp, a top-down compression-based tokeniser. They train language models with tokenisers from each algorithm, varying model size, vocabulary size, and domain (English-only vs. multilingual). Evaluated in bits-per-byte, the search procedure—rather than the objective—dominates: bottom-up tokenisers consistently achieve lower bits-per-byte across most settings. However, on BLiMP linguistic acceptability tasks, no consistent relationship exists between design choices and performance. The work offers concrete, principled guidance for tokeniser design in modern language models.

Paper Overview

Field: NLP Authors: Ahmetcan Yavuz, Clara Meister, Tiago Pimentel Published: 2026-09-16 arXiv: 2609.19145

Abstract

Two dominant tokenisation algorithms are used by modern language models: byte-pair encoding (BPE) and UnigramLM. These differ along two orthogonal axes: their optimisation objective (compression vs. log-likelihood) and their search procedure (bottom-up merging vs. top-down pruning). Existing comparisons confound these axes, making it unclear whether their observed differences stem from *what* is being optimised vs. *how* it is being optimised. The authors disentangle the two by introducing two new tokenisation algorithms that complete this 2x2 design space: BottomUpLL, a bottom-up likelihood-based tokeniser, and TopDownComp, a top-down compression-based tokeniser.

They train language models with tokenisers produced by each algorithm, varying model size, vocabulary sizes, and domain (English-only vs. multilingual).

Key Findings

Search procedure dominates the objective: Evaluated in bits-per-byte, bottom-up tokenisers consistently achieve lower bits-per-byte in most settings, regardless of whether the objective is compression or likelihood.
No consistent link to linguistic evaluation: On BLiMP tasks, there is no consistent correlation between tokeniser design choices and performance.
Practical implication: The work decouples the effects of tokeniser design choices on language modeling performance, providing concrete guidance for more principled tokeniser construction.

---

*Auto-collected on 2026-09-18*