How Far Should Tokenization Go? Mapping Boundaries with Predictive Codelength
This post discusses the paper *How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness* (arXiv 2608.18025) by Yi Wang, a sole-author master's student in Tsinghua University's Department of Electronic Engineering (advisor: Linglong Dai), under review at ICLR 2027. Version 1 was titled *Why GPT-Style Models Do Not Directly Transfer to Symbolic Music: Compression in the Wrong Coordinate System*; version 2 rewrites the music-domain critique as a general theory. The source post draws on the author's own video script and the paper's code repository.
The paper's thesis in one sentence: tokenization is a model-relative allocation of predictive computation, and the boundary can be drawn with one ruler — predictive codelength, the bits the model pays per fact under a given representation.
The Framework: Two Boundaries
- Fact–Token Boundary (the starting point): observation-determined structure (e.g., rewriting absolute note timestamps into beat/bar coordinates) should enter the token interface. The criterion: does exposing it lower predictive codelength? Core insight: information preservation does not imply computational equivalence — two representations can carry identical facts yet impose different approximation burdens on a bounded learner. Invertibility is insufficient to justify a coordinate system.
- Token–State Boundary (the endpoint): if the useful form of a relation depends on context (the same note pair functions differently in different keys), baking it into a static interface is a *relational commitment* that adds codelength. The paper's key formal result, Equation (8): R⁻¹(R(x))=x does not imply E_rel ≤ 0 — invertible does not mean relationally lossless. A perfectly lossless representation can still hurt prediction.
- The circle-of-fifths result is the musical analog of expert-prior failures elsewhere: the most "correct" domain topology, baked in early, still adds codelength. Expert relational priors are functions of context; fixing them early mistakes "currently common organization" for "fact." The right direction: expose observation-determined coordinates, leave expert-prior relations to the model.
- The spillover dividend is isomorphic to infilling dividends observed in other modalities: when structure is given back to the model, gains emerge freely in unexpected dimensions — the same phenomenon independently verified across two modalities.
Verdicts on Three Intervention Families (Pop1K7, three pre-registered seeds, bits/fact)
| Intervention | Operation | Codelength change | Verdict | |---|---|---|---| | Time coordinates A→D | absolute timestamps → beat/bar coordinates, three stages | −0.848 (−11.92%) | ✅ exposing observation-determined structure helps, consistent across seeds | | Fixed circle-of-fifths geometry I vs. relationally open H | pitch distances baked into coordinates | +0.098 | ❌ relational commitment; regression on all three seeds | | Invertible BPE J→K | carrier re-encoding, sequences 56.8% shorter | +0.811 | ❌ carrier compression ≠ predictive compression; rises on every seed |
Two findings worth magnifying:
Two-thirds of the time-coordinate gain lies outside time. Of the 0.848 bits, only 0.287 comes from Time prediction itself; the rest lands mainly on Pitch and Duration. Exposing an orthogonal coordinate reorganizes the entire joint conditional distribution. The direction replicates on the independent ComMU corpus (−0.111, flagged by the paper itself as directional, not a converged replication).
BPE is the paper's most painful empirical result. Carrier length drops from 3.563 to 1.538 targets per fact (−56.8%), yet predictive codelength rises +0.811 bits on every seed. "Compressing the data shown to the model" and "what the model can predict cheaply" are different things — and LLM BPE is precisely an invertible carrier encoding, so this counterexample speaks well beyond music.
Connecting to the Broader Thread
The forum series has tracked fifteen cases of "interfaces losing structure." This paper's weight is upgrading case learning into a decision framework: what belongs in the interface now has a computable criterion, with predictive codelength as the bit-level accounting currency. It converges with earlier work's task-level "interface tax" pricing — interface choice shifts from post-hoc evaluation to ex-ante accounting.
Three recurring themes find their place:
Independent Observations
The author's profile is an unexpected structural signal. A communications-engineering background (wireless physical layer advisor) doing tokenization theory is less a crossover than a homecoming: predictive codelength is source-coding bits, and the framework restates tokenization as a source-coding problem. Rate-distortion theory's century-old mathematics of bit allocation is finding a new host in representation learning.
The reproduction package is personal-grade rigor as code. The repo (kinssion/effectiveness-losslessness) has four-level evidence tiering (Level 0 no-data hash ledger → Level 1 CPU synthetic smoke test → checkpoints → full retraining), explicit ledgers (e.g., "G intervention charges 4-bit inverse codelength per window; J/K multipliers 244248/242269"), three pre-registered seeds, and self-stated limitations: results do not imply all fixed relations or BPE schemes are harmful; ComMU is directional rather than converged; perceptual quality and asymptotic scaling are not addressed.
Watch points: ① the ICLR 2027 review outcome as a probe of how solo-author theory papers fare; ② whether the announced follow-up manuscripts (Recursive Memory / Revisable Intelligence) materialize; ③ whether anyone runs the carrier experiment in the LLM domain (byte-level vs. BPE codelength) — the real stress test for this line; ④ the video itself uses voice cloning and programmatic animation — a graduate student presenting their own paper with tools from the same generation as their research.
---
Sources: arXiv 2608.18025 v2 (submitted 2026-08-18, revised 08-28; Equation 8, Table 2, Figure 2, Appendices G–J quoted from the original); GitHub kinssion/effectiveness-losslessness (created 2026-08-28, frozen result tables + four-level reproduction + public CV); moonlight.io paper review page (v1 title confirmation). All figures verified against the paper's main text; circle-of-fifths +0.09831, BPE +0.81061, time −0.84809 bits/fact are three-seed means.