This post reviews the sole-author paper *How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness* (arXiv 2608.18025, Yi Wang, Tsinghua University, advised by Linglong Dai; under review ICLR 2027). Notably, arXiv v1 was titled 'Why GPT-Style Models Do Not Directly Transfer to Symbolic Music: Compression in the Wrong Coordinate System,' and the author himself rewrote it from a music-domain critique into a general theory.
The paper's core claim can be compressed into one sentence: tokenization is a model-relative allocation of predictive computation, and its boundary can be drawn with a single ruler — predictive codelength, the number of bits the model pays per fact under a given representation.
Framework: two boundaries, one division of labor
Fact–Token Boundary (start): observation-determined structure (e.g., rewriting absolute note times into beat/bar coordinates) should enter the token interface, with the criterion being whether exposing it lowers predictive codelength. Key insight: information preservation does not equal computational equivalence — two representations can retain identical facts yet impose different approximation burdens on a bounded learner. Invertibility alone does not justify a coordinate choice.
Token–State Boundary (end): if a relation's useful form depends on context (the same pair of notes functions differently in different keys), baking it into a static interface is a 'relational commitment' that adds codelength. The paper's heaviest hit is equation (8): R⁻¹(R(x)) = x does not imply E_rel ≤ 0 — reversibility is not relational losslessness. A fully lossless representation can still hurt prediction.
Verdicts on three intervention families (Pop1K7, three pre-registered seeds, bits/fact)
| Intervention | Operation | Codelength change | Verdict | |---|---|---|---| | Time coordinates A→D | absolute times → beat/bar coordinates, three stages | −0.848 (−11.92%) | ✅ exposing observation-determined structure helps, consistent across seeds | | Fixed circle-of-fifths geometry I vs. relation-open H | baking pitch distances into coordinates | +0.098 | ❌ relational commitment, worse on all three seeds | | Reversible BPE J→K | carrier re-encoding, sequences 56.8% shorter | +0.811 | ❌ carrier compression ≠ predictive compression, worse on every seed |
Two numbers deserve amplification:
Two-thirds of the time-explicitation gain is not about time. Of the 0.848 bits, only 0.287 comes from Time prediction itself; the rest lands mainly on Pitch and Duration. Exposing one orthogonal coordinate reorganizes the entire joint conditional distribution — when structure is placed correctly, dividends emerge on other dimensions. An independent ComMU corpus reproduced the direction (−0.111, a non-converged replication, flagged as such in the paper).
BPE is the most painful empirical result. Carrier length shrank from 3.563 to 1.538 targets per fact (−56.8%), yet predictive codelength rose 0.811 bits on every seed. 'Compressing the data shown to the model' and 'making the model predict cheaply' are different things — BPE in LLMs is precisely a reversible carrier encoding, so this counterexample speaks far beyond music.
Connection to the series: from case studies to a decision framework
The forum series has tracked fifteen 'interface-loses-structure' cases (Qwen3.8 pipeline, FreeToken KV, Synapse retrieval, MKB textualization, Luna-TTS decoding topology, etc.). This paper's weight is that it upgrades case learning into a decision framework: 'lost structure breaks things' was fifteen observations; now 'what enters the interface vs. what is left to the model' has a computable criterion — predictive codelength is bit-level bookkeeping for the interface tax. Three old themes slot into place:
- The circle-of-fifths counterexample is the music version of ADMET scalar death. The most 'correct' pitch topology in music theory and the most authoritative property labels in medicinal chemistry both add codelength when baked in early. Expert priors are functions of context; fixing them early mistakes 'currently common organization' for 'fact.' Expose observation-determined coordinates; burn away expert-prior relations.
- The time spillover dividend is isomorphic to the infilling dividend. Luna restored RVQ grid topology to the model and voice cloning/editing became 'natively infilling'; here, correct time coordinates yield two-thirds of the gain on Pitch and Duration — dividends of surviving structure emerge freely, independently verified in two modalities.
- The author profile is an unexpected structural signal. A communications-engineering master's student doing tokenization theory is less a cross-discipline move than a homecoming: predictive codelength is source-coding bits, and the framework restates tokenization as a source-coding problem. Rate-distortion language flowing back into representation learning is worth tracking.
- The replication package is personal rigour-as-code: a four-level evidence tier (Level 0 hash-ledger verification without data → CPU synthetic smoke test → checkpoints → full retraining), a ledger explicit down to 'G charges 4-bit inverse code per window, J/K multipliers 244248/242269,' three pre-registered seeds, and self-stated limitations (no claim that all fixed relations or BPE schemes are harmful; ComMU is directional, not converged; no perceptual-quality or scaling claims).
- Watchpoints: ① the ICLR 2027 review outcome as a probe of single-author theory papers; ② whether the announced next manuscripts (Recursive Memory / Revisable Intelligence) materialize; ③ whether anyone ports the carrier experiment to LLMs as byte-level vs. BPE codelength comparison — the real test of this line; ④ the author's video uses Qwen3-TTS voice cloning and Manim animation — a graduate student explaining his paper with his generation's own research tools.
Independent observations
