Tokenizers Secretly Supervise Your Output: A Fundamental Problem 90% of Papers Ignore

Tokenizers Secretly Supervise Your Output: A Fundamental Problem 90% of Papers Ignore

Imagine evaluating two LLMs on math reasoning. Model A answers "347 + 231 = ?" correctly; Model B does not. You conclude Model A is better at math. But what if the two models use different tokenizers—if Model A's tokenizer encodes "578" as a single token while Model B's splits it into "5", "7", "8", then Model B is solving a harder task: producing three tokens across multiple forward passes instead of one.

In September 2026, Tanja Baeumel et al. from TU Darmstadt published a paper on arXiv making an unsettling claim: the tokenizer is not just input preprocessing—it is also output supervision, determining the supervision signal the model receives during training.

Paper: https://arxiv.org/abs/2609.01386

A Choice Defaulted to "Irrelevant"

In standard NLP pipelines, tokenizer choice is treated like code indentation style—acknowledged to matter, rarely treated as a key variable. The authors surveyed 120 recent ACL numerical reasoning papers:

• Only about 10% report the models' number tokenization scheme
• About 69% compare models across different tokenization schemes without saying so

This means many "Model A beats Model B" conclusions may reflect differences in task definition, not model capability.

How Does the Tokenizer Become a "Supervision Signal"?

When a model generates "578":

• If it is one token, the model needs "one-shot answer" capability: encode the whole answer internally, then decode it at once.
• If it is three tokens, the model must output "5", then "7" conditioned on "5", then "8" conditioned on "57"—a step-by-step reasoning task requiring maintenance of intermediate states.

The paper's core insight: the output-side granularity of the tokenizer determines what the model is supervised on during training. A single-token signal asks "is this token correct?"; a multi-token signal asks "is this token correct, then the next, then…". These supervision signals train different capabilities.

A Clever Decoupling Experiment

The authors decoupled input and output tokenization: inputs tokenized one way (e.g., digit-by-digit), outputs another (e.g., multi-digit), and vice versa.

If tokenization mainly affected input representation, changing input tokenization should matter most; if it affected output supervision, changing output tokenization should matter most.

The result was clear: output tokenization determined training dynamics, internal representations, and final performance, while input tokenization had almost no effect—directly contradicting the "tokenizer = input preprocessing" view.

What This Means

1. Cross-model comparisons may be unfair. When you say "GPT-4's math beats Claude's", differing tokenizers may mean you are comparing task difficulty, not capability. The paper notes tokenization-induced performance gaps on numerical reasoning can exceed 20 percentage points. 2. Distillation needs care. If teacher and student use different tokenizers, you distill not just knowledge but a "task definition". The student may learn "how to do math as defined by this tokenization", not "how to do math". 3. Tokenizer choice is a core design decision, not an implementation detail. The authors recommend all comparison papers report tokenizer information, especially for tokenization-sensitive domains like numbers and code.

A Deeper Question: What Are We Actually Evaluating?

The post's author draws a broader lesson: when we say "Model A beats Model B on task X", we are often evaluating "model + tokenizer + prompt template + decoding strategy + evaluation script" as a whole. The tokenizer is an extreme case because its effect is fundamental—it changes the training-time supervision signal—but similar issues exist elsewhere.

Evaluation blind spots often lie not where we cannot see, but where we see yet assume "irrelevant".

This paper joins a lineage of LLM evaluation blind-spot work:

• Omission Blindness: LLM judges detect what is said, not what is omitted
• MIST: attention and causal contribution are nearly uncorrelated
• Aspire: model self-assessment barely tracks real capability
• Calibrated Enough to Know: models can often judge "this is unpredictable", but the action gate ignores that judgment

Plus this one: output tokenization granularity determines the supervision signal, yet 90% of papers don't report tokenization. The common theme: the components defaulted to "irrelevant" in evaluation are often the most impactful.

Limitations and Open Questions

The experiments focus on numerical reasoning; generalization to code generation or natural language inference needs verification. The paper studies tokenization's effect on output supervision only—input tokenization's (smaller) effect is not zero.

Still, as work that exposes a systematic blind spot, it is convincing: it need not prove tokenization causes all problems, only that it is an important ignored variable—which it does.

Conclusion

Next time you see "Model A beats Model B by X percentage points", ask first: do they use the same tokenizer?

If not, how much of that X reflects model capability, and how much reflects task difficulty defined by the tokenizer? You probably won't get an answer—because 90% of papers won't tell you.

---

Paper: https://arxiv.org/abs/2609.01386 Authors: Tanja Baeumel, Josef van Genabith, Simon Ostermann Institution: TU Darmstadt