Latent Core Tokenizer: A Tokenizer That Understands Language Structure Before Compressing

Have you ever considered that when GPT chats with you, it doesn't see the same text you do?

You read "today's weather is great"; the model reads a sequence of tokens. Splitting human text into machine-consumable units is what a tokenizer does — and this seemingly trivial preprocessing step directly shapes how much information a language model can see and how well it learns.

A recent large-scale paper covering 104 languages and a 200K vocabulary proposes a counterintuitive conclusion: a tokenizer should not just maximize compression; it should first understand linguistic structure, then decide how to compress. That is the Latent Core Tokenizer (LCT).

The Old Problem: Compression ≠ Fairness

Mainstream BPE (Byte Pair Encoding) works bluntly: count the most frequent adjacent character pairs in the corpus, merge them into a token, and repeat until the vocabulary is full. Simple, and compression is high.

But the problem emerges in multilingual settings. In a vocabulary serving Chinese, English, and Swahili, BPE favors frequent English combinations because English data dominates; low-resource languages get chopped into more tokens for the same sentence. It's like a translation company giving the English department a full dictionary while other departments make do with pinyin.

Concrete numbers: with 104 languages and a 200K vocabulary, LCT achieves lower fertility (fewer average tokens per language) than BPE, Unigram, and Parity-Aware BPE, while scoring higher on MorphScore (better morphological structure preservation). On four multilingual downstream tasks, LCT beats BPE by 1.48 points, Unigram by 1.83, and Parity-Aware BPE by 2.00.

Core Innovation: Separating Structure Discovery from Vocabulary Building

LCT splits tokenization into two stages:

Stage 1: Latent Discovery. Three signals identify linguistic units worth preserving:

• Minimum Description Length (MDL): a segment that recurs and saves encoding length each time deserves its own token — like giving "ing" its own ID in English.
• Branching Entropy: if many different characters can follow a position, it's likely a morpheme boundary. "un-" can precede happy, fair, likely — high branching entropy marks it as a meaningful unit.
• Morphotactic Constraints: not all concatenations are valid. English "-s" attaches to nouns as a plural but not arbitrarily to verbs; language-specific constraints filter bad segmentations.

Stage 2: Surface Construction. LCT doesn't promote latent units directly into tokens. A "packer" selects which combinations deserve an embedding slot: frequent full words and common root+affix combos are merged first.

This separation is key: structure discovery decides which building blocks matter; surface packing decides which combinations deserve a slot. A productive root can influence vocabulary construction even if it ends up contained inside a whole-word token.

An Analogy: Library Cataloging

BPE is like shelving books by co-borrowing frequency: English bestsellers get prime shelf space; minority-language books get corners. LCT first hires linguists to analyze each language's internal structure — roots, affixes, common collocations — then allocates shelf space using that structure plus frequency. Structure guides allocation without dictating final placement.

Why Compression Rate ≠ Representation Quality

The paper's deepest finding: "compression alone does not predict representation quality."

Counterintuitive? Consider: splitting all Chinese into single characters gives perfect per-character compression but loses semantics — "grape" as one unit differs from two disconnected characters. Parity-Aware BPE optimizes compression (balancing token efficiency across languages) yet underperforms LCT downstream, because LCT also checks whether units correspond to meaningful linguistic structure.

Language Balance: Weighting, Not Quotas

LCT doesn't assign fixed vocabulary quotas per language. Instead it weights corpora — amplifying low-resource languages and shrinking high-resource ones — so each language contributes roughly equally to structure discovery. Final vocabulary allocation still follows frequency and efficiency, but discovery is no longer dominated by large languages.

Honest Limitations

• The three discovery signals (MDL, branching entropy, morphotactic constraints) require per-language tuning; despite the "language-agnostic" claim, morphotactic constraints need expert prior knowledge.
• 200K is a large vocabulary budget; LCT's advantage at smaller sizes (e.g., 50K) is untested.
• Gains of 1.48–2.00 points are statistically significant but may not be perceptible in practice; end-to-end evaluation is needed.

Still, the core insight stands: a tokenizer should not be a mere compression algorithm — it should be a language understanding system. While everyone watches parameter counts and training data, this piece of "infrastructure" is often overlooked. LCT reminds us: the world a model sees depends on the lens you give it.

---

Paper: Latent Core Tokenizer: Compress, but Meaningfully arXiv: 2610.12376 Authors: Felermino D. M. A. Ali, Millicent Ochieng, Ogbemi Ekwejunor-Etchie, et al.