Tokenizers Secretly Supervise Your Output: A Fundamental Problem Ignored by 90% of Papers

Suppose you are evaluating two LLMs on mathematical reasoning. Model A answers "347 + 231 = ?" correctly; Model B gets it wrong. You conclude Model A is better at math.

But what if the two models use different tokenizers? If Model A's tokenizer encodes "578" as a single token while Model B's splits it into "5", "7", "8", Model B is actually solving a harder task—it must generate three tokens across multiple forward passes where Model A generates one.

In September 2026, Tanja Baeumel and colleagues at TU Darmstadt published a paper on arXiv making an unsettling claim: tokenization is not just input preprocessing—it is also output supervision, determining the supervision signal a model receives during training.

Paper: https://arxiv.org/abs/2609.01386

A Choice Defaulted to "Irrelevant"

In the standard NLP pipeline, tokenizer choice is treated like choosing an indentation style—acknowledged to have some effect, but rarely considered a key variable. The authors surveyed 120 recent "numerical reasoning" papers published in *ACL venues:

• Only about 10% reported how numbers were tokenized in their models
• About 69% compared models across different tokenization schemes without stating so

This means many "Model A beats Model B" conclusions may reflect differences in task definition, not model capability.

How Does a Tokenizer Become a "Supervision Signal"?

To understand the claim, consider how autoregressive language models work. When a model generates the answer "578":

• If the tokenizer encodes "578" as one token, the model outputs it in a single forward pass—it needs a "one-shot answer" capability.
• If "578" is split into "5", "7", "8", the model outputs "5", then "7" conditioned on "5", then "8" conditioned on "57". It needs a "step-by-step reasoning" capability.

These look like the same task but are fundamentally different. The former is a recognition task—the model encodes the whole answer in its internal state and decodes it at once. The latter is a reasoning task—the model maintains an intermediate state and makes sequential decisions.

The paper's core insight: the output-side granularity of the tokenizer determines what the model is supervised on during training. A single-token signal asks "is this token correct?"; a multi-token signal asks "is this token correct, then the next, then...". These signals train different capabilities.

A Clever Decoupling Experiment

To test this hypothesis, the authors designed an input–output tokenization decoupling experiment.

Normally, input and output share one tokenizer. But the authors let the input use one tokenization scheme (e.g., splitting numbers into single characters) and the output another (e.g., multi-digit chunks), and vice versa.

If tokenization matters via input representation, changing input tokenization should matter most; if it matters via output supervision, changing output tokenization should matter most.

The result was clear: output tokenization determined training dynamics, internal representations, and final performance, while input tokenization had almost no effect.

This directly contradicts the traditional view of the tokenizer as input preprocessing. The tokenizer's main effect is on the output side—it defines the supervision signal received in training.

What Does This Mean?

The finding has several far-reaching implications:

First, cross-model comparisons may be unfair. When you say "GPT-4 is better at math than Claude," if the models use different tokenizers, you may be comparing task difficulty rather than model capability. The paper notes that on numerical reasoning, different tokenization schemes cause performance gaps of over 20 percentage points.

Second, model distillation requires caution. If teacher and student models use different tokenizers, distillation transfers not only knowledge but also a "task definition". The student may learn not "how to do math" but "how to do math as defined by this tokenization".

Third, tokenizer choice should be treated as a core design decision, not an implementation detail. The paper recommends that all papers involving model comparisons report tokenizer information, especially in tokenization-sensitive domains like numbers and code.

A Deeper Question: What Are We Actually Evaluating?

This paper raises a broader question: when we say "Model A beats Model B on task X," what exactly are we evaluating?

Often we evaluate not "the model's capability" but the capability of the whole pipeline: "model + tokenizer + prompt template + decoding strategy + evaluation script". If any component differs, the comparison may be unfair.

Tokenization is an extreme case because its effect is foundational—it changes the supervision signal during training. But similar problems exist elsewhere: different prompt templates activate different capabilities, decoding strategies produce different output distributions, and evaluation scripts may define "correct answers" differently.

Blind spots in evaluation are often not where we cannot see, but where we see yet default to "irrelevant". The tokenizer is exactly such a component.

Another Case of the "Evaluation Blind Spot"

This paper joins a lineage of work on LLM evaluation blind spots:

• Omission Blindness: LLM judges detect what is said but not what is omitted
• MIST: attention and causal contribution are nearly uncorrelated
• Aspire: models' self-assessments barely correlate with real capability
• Calibrated Enough to Know: models know 90% of the time when something is unpredictable, but the action gate never consults the judgment module

The common theme: components of evaluation that are assumed "irrelevant" are often the most impactful—not because they are complex, but because they are so basic they get overlooked.

Limitations and Open Questions

The paper's experiments focus on numerical reasoning; whether the conclusions generalize to other tasks (code generation, natural language inference) remains to be verified. It studies tokenization's effect on output supervision but not comprehensively on input representation—though experiments show input tokenization matters less, that does not mean it has no effect.

But as a work that exposes a systemic blind spot, the paper is already powerful. It does not need to prove tokenization is the root of all problems—only that it is an overlooked important variable. On that, it delivers.

Conclusion

Next time you see "Model A beats Model B by X percentage points", ask one question first: do they use the same tokenizer?

If not, how much of that X points is model capability, and how much is task difficulty defined by tokenization?

You probably won't get an answer—because 90% of papers won't tell you.

---

Paper: https://arxiv.org/abs/2609.01386 Authors: Tanja Baeumel, Josef van Genabith, Simon Ostermann Affiliation: TU Darmstadt