Tokenizers Secretly Supervise Your Output: A Fundamental Problem Overlooked by 90% of Papers

Tokenizers Secretly Supervise Your Output: A Fundamental Problem Overlooked by 90% of Papers

Suppose you are evaluating the mathematical reasoning of two LLMs. Model A answers "347 + 231 = ?" correctly; Model B answers incorrectly. You conclude Model A has stronger math ability.

But what if the two models use different tokenizers? If Model A's tokenizer encodes "578" as a single token while Model B's splits it into "5", "7", "8", then Model B is actually solving a harder task—generating three tokens across successive forward passes, while Model A only needs one.

In September 2026, Tanja Baeumel et al. from TU Darmstadt published a paper on arXiv making an unsettling claim: tokenization is not just input preprocessing—it is also output supervision—it determines the supervision signal a model receives during training.

Paper link: https://arxiv.org/abs/2609.01386

A Choice Defaulted to "Irrelevant"

In standard NLP research practice, tokenizer choice is treated like indentation style—acknowledged to matter, but rarely considered a key variable. The authors surveyed 120 recent numerical reasoning papers published in *ACL venues:

• Only about 10% report the models' numerical tokenization scheme
• About 69% compare models across different tokenization schemes without stating so

This means many "Model A beats Model B" conclusions may reflect differences in task definition, not model capability.

How Does a Tokenizer Become a "Supervision Signal"?

To understand the claim, consider how autoregressive language models work.

When a model generates the answer "578":

• If the tokenizer encodes "578" as one token, the model outputs it in a single forward pass. It needs the ability to "give the answer at once."
• If the tokenizer splits "578" into "5", "7", "8", the model must first output "5", then output "7" conditioned on "5", then "8" conditioned on "57". It needs the ability to "reason step by step."

These two tasks look the same but are fundamentally different. The former is a recognition task—the model must encode the entire answer in its internal state and decode it at once. The latter is a reasoning task—the model must maintain an intermediate state and make a decision at each step based on prior outputs.

The paper's core insight: the output-side granularity of the tokenizer determines what the model is supervised on during training. A single token's supervision signal is "is this token correct"; multiple tokens' signal is "is this token correct, then the next, then...". These two signals train different capabilities.

A Clever Decoupling Experiment

To test the hypothesis, the authors designed an input-output tokenization decoupling experiment.

Normally input and output share the same tokenizer. But here, inputs used one tokenization scheme (e.g., digits split into single characters) while outputs used another (e.g., multi-digit tokens), and vice versa.

If tokenization affects "input representation", changing input tokenization should matter most; if it affects "output supervision", changing output tokenization should matter most.

The result was clear: output tokenization determines training dynamics, internal representations, and final performance, while input tokenization has almost no effect.

This directly contradicts the traditional view of the tokenizer as input preprocessing. The tokenizer's primary role is on the output side—it defines the supervision signal received during training.

What Does This Mean?

Several far-reaching implications:

First, cross-model comparisons may be unfair. When you say "GPT-4 is better at math than Claude," if the two use different tokenizers, you may be comparing task difficulty as defined by tokenizers, not model capability. The paper notes tokenization-induced performance differences on numerical reasoning can exceed 20 percentage points.

Second, distillation requires caution. If teacher and student models use different tokenizers, what is distilled includes not just knowledge but "task definition". The student may learn not "how to do math" but "how to do math as defined by this tokenization".

Third, tokenizer choice should be treated as a core design decision, not an implementation detail. The authors recommend that all papers involving model comparisons report tokenizer information, especially in tokenization-sensitive domains like numbers and code.

A Deeper Question: What Are We Actually Evaluating?

This paper evokes a broader question: when we say "Model A beats Model B on task X", what exactly are we evaluating?

Often we are not evaluating "model capability" but the combined system of "model + tokenizer + prompt template + decoding strategy + evaluation script". If any component differs, the comparison may be unfair.

Tokenization is an extreme case because its influence is fundamental—it changes the supervision signal during training. But similar issues exist elsewhere: different prompt templates can activate different capabilities, decoding strategies produce different output distributions, and evaluation scripts may define "correct answers" differently.

Blind spots in evaluation are often not in what we cannot see, but in what we see yet default to "irrelevant". Tokenization is exactly such a component.

Another Case in the "Evaluation Blind Spot" Series

This paper joins a lineage of work on LLM evaluation blind spots:

• Omission Blindness: LLM judges detect "what was said" but not "what was omitted"
• MIST: attention and causal contribution are nearly uncorrelated
• Aspire: model self-assessment barely correlates with true capability
• Calibrated Enough to Know: models can 90% judge "this is unpredictable", but the action gate never consults the judgment module

The common theme: components in the evaluation pipeline that are defaulted to "irrelevant" are often the most influential—not because they are complex, but because they are so basic that they get ignored.

Limitations and Open Questions

The paper's experiments are mainly on numerical reasoning tasks; whether the conclusions generalize to other tasks (code generation, natural language inference) remains to be validated. Also, the paper studies tokenization's effect on output supervision, not input representation—though experiments show input tokenization matters less, that doesn't mean it has no effect, only a smaller one.

But as a work "pointing out a systematic blind spot", this paper is already powerful. It doesn't need to prove tokenization is the root of all problems—only that it is an overlooked, significant variable. On that point, it succeeds.

Conclusion

Next time you see "Model A beats Model B by X percentage points", first ask: do they use the same tokenizer?

If not, how much of that X points difference is model capability, and how much is task difficulty defined by tokenization?

You probably won't get an answer—because 90% of papers won't tell you.

---

Paper link: https://arxiv.org/abs/2609.01386 Authors: Tanja Baeumel, Josef van Genabith, Simon Ostermann Institution: TU Darmstadt