Objective vs. Search: Decomposing What Makes a Good Tokeniser (arXiv 2609.19145)

Forum topic · 小凯 · 2026-09-18

Summary

A new arXiv paper (2609.19145) by Ahmetcan Yavuz, Clara Meister, and Tiago Pimentel disentangles two orthogonal design axes of tokenisation algorithms: the optimisation objective (compression vs. log-likelihood) and the search procedure (bottom-up merging vs. top-down pruning). The authors introduce two new tokenisers that complete the 2x2 design space: BottomUpLL, a bottom-up likelihood-based tokeniser, and TopDownComp, a top-down compression-based tokeniser. They train language models with tokenisers from each algorithm, varying model size, vocabulary size, and domain (English-only vs. multilingual). Evaluated in bits-per-byte, results show that the search procedure—not the optimisation objective—is the dominant factor: bottom-up tokenisers consistently achieve lower bits-per-byte across most settings. However, on the BLiMP linguistic acceptability benchmark, no consistent relationship emerges between design choices and performance. The work provides principled guidance for tokeniser design by separating what is optimised from how it is optimised.

Paper Overview

Field: NLP Authors: Ahmetcan Yavuz, Clara Meister, Tiago Pimentel Published: 2026-09-16 arXiv: 2609.19145

Introduction

Two dominant tokenisation algorithms are used by modern language models: byte-pair encoding (BPE) and UnigramLM. These differ along two orthogonal axes:

Optimisation objective: compression vs. log-likelihood
Search procedure: bottom-up merging vs. top-down pruning

Existing comparisons confound these axes, making it unclear whether observed performance differences stem from *what* is being optimised vs. *how* it is being optimised.

Method: Completing the 2x2 Design Space

The authors introduce two new tokenisation algorithms to decouple the two dimensions:

1. BottomUpLL — a bottom-up, likelihood-based tokeniser 2. TopDownComp — a top-down, compression-based tokeniser

Together with BPE (bottom-up compression) and UnigramLM (top-down likelihood), these complete the full 2x2 design space.

Experiments

Language models were trained with tokenisers produced by each algorithm, varying:

Model size
Vocabulary sizes
Domain (English-only vs. multilingual)

Findings

Bits-per-byte: The search procedure, not the optimisation objective, is the dominant factor. In most settings, bottom-up tokenisers consistently achieve lower bits-per-byte.
BLiMP: No consistent relationship exists between tokeniser design choices and linguistic acceptability performance.

Conclusion

The work decouples the effects of tokeniser design choices on language modelling performance, offering concrete guidance for more principled tokeniser construction.

--- *Auto-collected on 2026-09-18*